Predicting user's confidence during visual decision making

Smith, J., Legg, P., Kinsey, K. and Matovis, M. (2018) Predicting user's confidence during visual decision making. ACM Transactions on Interactive Intelligent Systems. Available from:

Full text not available from this repository

Publisher's URL:


People are not infallible consistent “oracles”: their confidence in decision-making may vary significantly be- tween tasks and over time. We have previously reported the benefits of using an interface and algorithms that explicitly captured and exploited users’ confidence: error rates were reduced by up to 50% for an indus- trial multi-class learning problem; and the number of interactions required in a design optimisation context was reduced by 33%. Having access to users’ confidence judgements could significantly benefit intelligent interactive systems in industry, in areas such as Intelligent Tutoring systems, and in healthcare. There are many reasons for wanting to capture information about confidence implicitly. Some are ergonomic, but others are more ‘social’ - such as wishing to understand (and possibly take account of) users’ cognitive state without interrupting them. We investigate the hypothesis that users’ confidence can be accurately predicted from measurements of their behaviour. Eye-tracking systems were used to capture users’ gaze patterns as they undertook a series of visual decision tasks, after each of which they reported their confidence on a 5-point Likert scale. Sub- sequently, predictive models were built using “conventional” Machine Learning approaches for numerical summary features derived from users’ behaviour. We also investigate the extent to which the deep learn- ing paradigm can reduce the need to design features specific to each application, by creating “gazemaps” – visual representations of the trajectories and durations of users’ gaze fixations – and then training deep convolutional networks on these images. Treating the prediction of user confidence as a two-class problem (confident/not confident), we attained classification accuracy of 88% for the scenario of new users on known tasks, and 87% for known users on new tasks. Considering the confidence as an ordinal variable, we produced regression models with a mean abso- lute error of ≈0.7 in both cases. Capturing just a simple subset of non-task-specific numerical features gave slightly worse, but still quite high accuracy (eg. MAE ≈1.0). Results obtained with gazemaps and convolu- tional networks are competitive, despite not having access to longer-term information about users and tasks, which was vital for the ‘summary’ feature sets. This suggests that the gazemap-based approach forms a vi- able, transferable, alternative to hand-crafting features for each different application. These results provide significant evidence to confirm our hypothesis, and offer a way of substantially improving many interactive artificial intelligence applications via the addition of cheap non-intrusive hardware and computationally cheap prediction algorithms.

Item Type:Article
Uncontrolled Keywords:human-centred machine learning, confidence
Faculty/Department:Faculty of Environment and Technology > Department of Computer Science and Creative Technologies
ID Code:34652
Deposited By: Professor J. Smith
Deposited On:26 Jan 2018 11:36
Last Modified:11 May 2018 14:22

Request a change to this item

Document Downloads

Total Document Downloads

More statistics for this item...