To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-making

被引：220
作者：
Buçinca Z. [1 ]
Malaya M.B. [2 ]
Gajos K.Z. [1 ]
机构：
[1] Harvard University, 33 Oxford St., Cambridge, 02138, MA
[2] Lodz University of Technology, ul. Stefana Zeromskiego 116, Lodz
来源：
Proceedings of the ACM on Human-Computer Interaction | 2021年 / 5卷 / CSCW1期
关键词：
artificial intelligence; cognition; explanations; trust;
D O I：
10.1145/3449287
中图分类号：
学科分类号：
摘要：
People supported by AI-powered decision support tools frequently overrely on the AI: they accept an AI's suggestion even when that suggestion is wrong. Adding explanations to the AI decisions does not appear to reduce the overreliance and some studies suggest that it might even increase it. Informed by the dual-process theory of cognition, we posit that people rarely engage analytically with each individual AI recommendation and explanation, and instead develop general heuristics about whether and when to follow the AI suggestions. Building on prior research on medical decision-making, we designed three cognitive forcing interventions to compel people to engage more thoughtfully with the AI-generated explanations. We conducted an experiment (N=199), in which we compared our three cognitive forcing designs to two simple explainable AI approaches and to a no-AI baseline. The results demonstrate that cognitive forcing significantly reduced overreliance compared to the simple explainable AI approaches. However, there was a trade-off: people assigned the least favorable subjective ratings to the designs that reduced the overreliance the most. To audit our work for intervention-generated inequalities, we investigated whether our interventions benefited equally people with different levels of Need for Cognition (i.e., motivation to engage in effortful mental activities). Our results show that, on average, cognitive forcing interventions benefited participants higher in Need for Cognition more. Our research suggests that human cognitive motivation moderates the effectiveness of explainable AI solutions. © 2021 ACM.
引用
收藏
相关论文
共 67 条
  • [1] Bansal G., Nushi B., Kamar E., Lasecki W.S., SWeld D.S., Horvitz E., Beyond accuracy: The role of mental models in human-AI team performance, Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, 7, pp. 2-11, (2019)
  • [2] Bansal G., Wu T., Zhu J., Fok R., Nushi B., Kamar E., Tulio Ribeiro M., Weld D.S., Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance, Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI '21), pp. 1-16, (2021)
  • [3] Barr D.J., Levy R., Scheepers C., Tily H.J., Random effects structure for confirmatory hypothesis testing: Keep it maximal, Journal of memory and language, 68, 3, pp. 255-278, (2013)
  • [4] Berner E.S., Graber M.L., Overconfidence as a cause of diagnostic error in medicine, The American journal of medicine, 121, 5, pp. S2-S23, (2008)
  • [5] Bhatt U., Weller A., Moura F.J.M., Evaluating and Aggregating Feature-based Model Explanations, Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20, pp. 3016-3022, (2020)
  • [6] Bornstein B.H., Christine Emler A., Rationality in medical decision making: a review of the literature on doctors' decision-making biases, Journal of evaluation in clinical practice, 7, 2, pp. 97-107, (2001)
  • [7] Bucinca Z., Lin P., Gajos K.Z., Glassman E.L., Proxy Tasks and Subjective Measures Can Be Misleading in Evaluating Explainable AI Systems, Proceedings of the 25th International Conference on Intelligent User Interfaces (IUI '20), (2020)
  • [8] Buolamwini J., Gebru T., Gender shades: Intersectional accuracy disparities in commercial gender classification, Conference on fairness, accountability and transparency, pp. 77-91, (2018)
  • [9] Bussone A., Stumpf S., O'Sullivan D., The role of explanations on trust and reliance in clinical decision support systems, 2015 international conference on healthcare informatics, pp. 160-169, (2015)
  • [10] Cacioppo J.T., Petty R.E., The need for cognition, Journal of Personality and Social Psychology, 42, 1, pp. 116-131, (1982)
1234567