Assessing Field Reliability of Forensic Decision Making in Criminal Court

The current study evaluated field reliability of forensic judgments in a state circuit court setting. Utilizing panels of three independently appointed examiners, Hawaii's system for evaluating mentally ill defendants charged with felonies provides a naturalistic laboratory for investigating decision-making processes in examiners and judges. The study focused on competency to stand trial (CST), criminal responsibility (NGRI), and postacquittal conditional release (CR) using a methodology adopted from clinical medicine and natural sciences. The entire adjudicative process from examination to judicial determination is examined as a forensic classification model. Examiner agreement and judicial consensus for 450 forensic reports were examined to assess performance in a non-crossed-data measurement design. Reliability of psycholegal constructs (CST, NGRI, postacquittal CR) was assessed using a novel reliability estimator (Krippendorff's alpha: KALPHA). This is the first study in the forensic mental health literature to examine interrater agreement for postacquittal conditional release decision making. Findings revealed adequate performance for CST, poor performance for CR, and marginal performance for NGRI decisions. Judges demonstrated independence in the presence of panel disagreements. Across the board, examiners demonstrated high levels of inconsistency in judgments. Factors associated with poor reliability, including task complexity, are discussed. The findings raise concerns about the overall quality of psycholegal decision making in criminal proceedings, especially in NGRI and CR determinations.