Dorfman-Berbaum-Metz method for statistical analysis of multireader, multimodality receiver operating characteristic data: Validation with computer simulation

Rationale and Objectives. The authors examined the relationship between the critical P value (alpha) and the empirical type I error rate when using the Dorfman-Berbaum-Metz (DBM) method for analysis of variance in multireader, multimodality receiver operating characteristic (ROC) data. Methods. The authors developed a linear mixed-effect model to generate continuous, normally distributed random decision variables containing multiple sources (components) of variation. A range of magnitudes for these variance components was used to simulate experiments in which multiple readers (three or five) read images obtained with two modalities from the same set of cases with no re-reading. Three binormal population ROC curves, with areas of 0.962, 0.855, and 0.702, were included. Case-sample sizes ranged from 50 to 400, and either 50% or 10% of cases were actually positive. For each experiment, 2,000 data sets were analyzed by the computer program, and the proportion of 2,000 modality differences that was found to be statistically significant at an alpha level of .05 was tabulated. Results. The test for modality difference performed well for the low and intermediate ROC curves, even with small case samples. For the high ROC curve, the small-sample results were conservative. No relationship between observed type I error rate and the magnitude of data correlation was evident. Conclusion. For typical ROC curves, the DBM method is robust in testing for modality effects in the null case, given a sufficient sample size. Instructions for obtaining a free copy of the software are given.