Analysis of diagnostic confidence and diagnostic accuracy: a unified framework

Diagnostic confidence has been used as a measure of diagnostic efficacy, but this measure in isolation fails to take into account incorrect diagnoses. Conventional analytical approaches of diagnostic confidence ignore associated diagnostic accuracy. To address this limitation, we introduce a unifying framework which incorporates diagnostic confidence, changes in diagnoses and ultimate accuracy. The framework is illustrated using data from a study in which 62 patients with acute abdominal pain prospectively underwent CT. Admitting surgeons documented their diagnoses and graded their diagnostic confidences (on a 5-point scale) on admission and again after CT. Our approach, unlike conventional analyses, incorporates knowledge of final diagnoses, obtained from surgery or 6 months follow up, in assessing the impact of the test (on a 9-point scale). Changes in pre- and post-CT confidence scores were assessed by the one-sample t-test comparing against zero change, with the test statistic acting as a standardized quantity allowing comparison between our and conventional methodological approaches. Overall, 52% (32/62) of patients were misdiagnosed on admission and 19% (12/62) had incorrect post-CT diagnoses. Diagnostic confidence following CT increased significantly compared with pre-CT confidence on applying both analytical methods, although the level of statistical significance was less marked using our approach. Mean (95% confidence interval) increase in confidence under conventional analysis was 1.32 (1.03, 1.62), with standardized score t=8.90 [p<0.0001], whereas our method yielded 0.69 (0.25, 1.13), with standardized score t=3.12 [p=0.003]. Although both analytical methods led to the same inference regarding the efficacy of CT in the illustrative case study presented, they differed somewhat in degree. it is conceivable that disparate conclusions may emerge in other studies and circumstances. Failure to take adequate account of incorrect diagnoses is potentially misleading. We suggest that a comprehensive analysis of diagnostic confidence requires the incorporation of diagnostic accuracy.