Methods to Evaluate Temporal Cognitive Biases in Machine Learning Prediction Models

When asked to rank or rate a list of items, humans are often affected by cognitive biases, which may lead to inconsistent decisions over time. These inconsistencies become part of machine learning prediction algorithms trained on human judgments, leading to misalignment and consequently affecting the metrics used to evaluate their correctness. In this paper, we propose new accuracy metrics, built upon commonly used statistics- and decision support-based metrics. Each of these metrics is designed to address the varying nature of human judgment and to evaluate the importance of decisions that change over time due to cognitive biases.