Assessing the confidence-accuracy relationship in risk assessments using the START: introducing calibration analysis

A risk assessor's confidence has been shown to influence both the rater as well as those evaluating the risk assessment. It is important to consider the impact of confidence on accuracy in the risk assessment field given the significant implications of risk assessments for the assessed (e.g. available treatment options vs. restrictions of liberties) and the safety of the public. While prior research in the risk assessment field has used correlations to investigate the confidence-accuracy (C-A) relationship, a number of fields in psychology have introduced additional techniques, including calibration analysis, to understand this relationship. In this study, we examined the C-A relationship across various adverse clinical outcomes using the Short-Term Assessment of Risk and Treatability (START) and compared and contrasted the C-A relationship using conventional methods (i.e. correlations/ROC analysis) and calibration. Raters completed START assessments for a sample of 106 civil psychiatric inpatients. Overall, calibration provided greater detail into the C-A relationship compared to correlations/ROC analysis. Our results also suggested that the C-A relationship varied as a function of the outcome assessed (e.g. violence, substance abuse, unauthorized leave). These results provide insights into the C-A relationship in medico-legal settings and can inform best practices for risk assessment training and implementation.