To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-making

被引：291
作者：
Buçinca Z. [1 ]
Malaya M.B. [2 ]
Gajos K.Z. [1 ]
机构：
[1] Harvard University, 33 Oxford St., Cambridge, 02138, MA
[2] Lodz University of Technology, ul. Stefana Zeromskiego 116, Lodz
来源：
Proceedings of the ACM on Human-Computer Interaction | 2021年 / 5卷 / CSCW1期
关键词：
artificial intelligence; cognition; explanations; trust;
D O I：
10.1145/3449287
中图分类号：
学科分类号：
摘要：
People supported by AI-powered decision support tools frequently overrely on the AI: they accept an AI's suggestion even when that suggestion is wrong. Adding explanations to the AI decisions does not appear to reduce the overreliance and some studies suggest that it might even increase it. Informed by the dual-process theory of cognition, we posit that people rarely engage analytically with each individual AI recommendation and explanation, and instead develop general heuristics about whether and when to follow the AI suggestions. Building on prior research on medical decision-making, we designed three cognitive forcing interventions to compel people to engage more thoughtfully with the AI-generated explanations. We conducted an experiment (N=199), in which we compared our three cognitive forcing designs to two simple explainable AI approaches and to a no-AI baseline. The results demonstrate that cognitive forcing significantly reduced overreliance compared to the simple explainable AI approaches. However, there was a trade-off: people assigned the least favorable subjective ratings to the designs that reduced the overreliance the most. To audit our work for intervention-generated inequalities, we investigated whether our interventions benefited equally people with different levels of Need for Cognition (i.e., motivation to engage in effortful mental activities). Our results show that, on average, cognitive forcing interventions benefited participants higher in Need for Cognition more. Our research suggests that human cognitive motivation moderates the effectiveness of explainable AI solutions. © 2021 ACM.
引用
收藏
相关论文
共 67 条
[51]  
Parush A., Ahuvia S., Erev I., Degradation in spatial knowledge acquisition when using automatic navigation systems, International conference on spatial information theory, pp. 238-254, (2007)
[52]  
Petty R.E., Cacioppo J.T., The Elaboration Likelihood Model of Persuasion, Communication and Persuasion, 19, pp. 1-24, (1986)
[53]  
Pop V.L., Shrewsbury A., Durso F.T., Individual differences in the calibration of trust in automation, Human factors, 57, 4, pp. 545-556, (2015)
[54]  
Deborah Raji I., Smart A., White R.N., Mitchell M., Gebru T., Hutchinson B., Smith-Loud J., Theron D., Barnes P., Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing, Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency (Barcelona, Spain) (FAT '20), pp. 33-44, (2020)
[55]  
Sherbino J., Kulasegaram K., Howey E., Norman G., Ineffectiveness of cognitive forcing strategies to reduce biases in diagnostic reasoning: a controlled trial, Canadian Journal of Emergency Medicine, 16, 1, pp. 34-40, (2014)
[56]  
Sicilia M., Ruiz S., Munuera J.L., Effects of interactivity in a web site: The moderating effect of need for cognition, Journal of advertising, 34, 3, pp. 31-44, (2005)
[57]  
Spilke J., Piepho H.P., Hu X., Analysis of unbalanced data by mixed linear models using the MIXED procedure of the SAS system, Journal of Agronomy and crop science, 191, 1, pp. 47-54, (2005)
[58]  
Tuten T.L., Bosnjak M., Understanding differences in web usage: The role of need for cognition and the five factor model of personality, Social Behavior and Personality: an international journal, 29, 4, pp. 391-398, (2001)
[59]  
Vaccaro M., Waldo J., The effects of mixing machine learning and human judgment, Commun. ACM, 62, 11, pp. 104-110, (2019)
[60]  
Veinot T.C., Mitchell H., Ancker J.S., Good intentions are not enough: how informatics interventions can worsen inequality, Journal of the American Medical Informatics Association, 25, 8, pp. 1080-1088, (2018)
1234567