What's the Control in Studies Measuring the Effect of Computer-Aided Detection (CAD) on Observer Performance?

Rationale and Objectives: The goal of many multiple-observer computer-aided detection (CADe) studies is to estimate the change in observers' diagnostic performance with CADe from their unaided performance. A key issue in these studies is the method for estimating the observers' unaided performance. The crossover design is considered the most valid. The sequential design takes less time and is less expensive but may be biased. We conducted a study to investigate the differences between these two designs. Materials and Methods: Data from two large CADe studies using both types of unaided reads were analyzed. The first study involved three radiologists examining the chest x-rays of 200 patients for lung nodules. The second study involved 19 observers interpreting the computed tomography colonography images of 100 patients for polyps. Observers' sensitivity, specificity, and receiver operating characteristic areas were estimated while unaided in both designs and compared to their accuracy with CADe. Bias, inter-observer variability, and correlations between unaided and aided results were assessed. Results: Observers tend to perform better while unaided in the sequential design than while unaided in the crossover design, but the differences are small. The inter-observer variability is larger in the sequential design. The correlations between unaided and aided results are larger in the sequential design. 95% CIs for the change with CADe are narrower with the sequential design. Conclusion: The estimated effect of CADe on observer performance is similar regardless of the study design. Use of the sequential design may save investigators time and resources.