Predicting user confidence during visual decision making

Smith, J., Legg, P., Kinsey, K. and Matovis, M. (2018) Predicting user confidence during visual decision making. ACM Transactions on Interactive Intelligent Systems, 8 (2). ISSN 2160-6455 Available from:

PDF - Accepted Version
Available under License All Rights Reserved.


Publisher's URL:


People are not infallible consistent “oracles”: their confidence in decision-making may vary significantly be- tween tasks and over time. We have previously reported the benefits of using an interface and algorithms that explicitly captured and exploited users’ confidence: error rates were reduced by up to 50% for an indus- trial multi-class learning problem; and the number of interactions required in a design optimisation context was reduced by 33%. Having access to users’ confidence judgements could significantly benefit intelligent interactive systems in industry, in areas such as Intelligent Tutoring systems, and in healthcare. There are many reasons for wanting to capture information about confidence implicitly. Some are ergonomic, but others are more ‘social’ - such as wishing to understand (and possibly take account of) users’ cognitive state without interrupting them. We investigate the hypothesis that users’ confidence can be accurately predicted from measurements of their behaviour. Eye-tracking systems were used to capture users’ gaze patterns as they undertook a series of visual decision tasks, after each of which they reported their confidence on a 5-point Likert scale. Sub- sequently, predictive models were built using “conventional” Machine Learning approaches for numerical summary features derived from users’ behaviour. We also investigate the extent to which the deep learn- ing paradigm can reduce the need to design features specific to each application, by creating “gazemaps” – visual representations of the trajectories and durations of users’ gaze fixations – and then training deep convolutional networks on these images. Treating the prediction of user confidence as a two-class problem (confident/not confident), we attained classification accuracy of 88% for the scenario of new users on known tasks, and 87% for known users on new tasks. Considering the confidence as an ordinal variable, we produced regression models with a mean abso- lute error of ≈0.7 in both cases. Capturing just a simple subset of non-task-specific numerical features gave slightly worse, but still quite high accuracy (eg. MAE ≈1.0). Results obtained with gazemaps and convolu- tional networks are competitive, despite not having access to longer-term information about users and tasks, which was vital for the ‘summary’ feature sets. This suggests that the gazemap-based approach forms a vi- able, transferable, alternative to hand-crafting features for each different application. These results provide significant evidence to confirm our hypothesis, and offer a way of substantially improving many interactive artificial intelligence applications via the addition of cheap non-intrusive hardware and computationally cheap prediction algorithms.

Item Type:Article
Additional Information:© ACM, 2018. This is the author's version of the work. It is posted here by permission of ACM for your personal use. Not for redistribution. The definitive version was published in ACM Transactions on Interactive Intelligent Systems, 8 (2), June 2018
Uncontrolled Keywords:human-centred machine learning, confidence
Faculty/Department:Faculty of Environment and Technology > Department of Computer Science and Creative Technologies
ID Code:34652
Deposited By: Professor J. Smith
Deposited On:26 Jan 2018 11:36
Last Modified:22 Jun 2018 12:37

Request a change to this item

Total Document Downloads in Past 12 Months

Document Downloads

Total Document Downloads

More statistics for this item...