Multiple-reader studies, digital mammography, computer-aided diagnosis - and the Holy Grail of imaging physics (II)

The metaphor of the Holy Grail is used here to refer to the classic and elusive problem in medical imaging of predicting the ranking of the clinical performance of competing imaging modalities from the ranking obtained from physical laboratory measurements and signal-detection analysis, or from simple phantom studies. In recent years a number of investigators have demonstrated the ability to predict the ranking of systems in simple phantom studies from the physical laboratory measurements and signal-detection theory. The ability to extrapolate from this point to clinical performance has been complicated by both the complexity of clinical images and the masking effect of reader variability. We show how the use of the multiple-reader, multiple-case (MRMC) ROC paradigm and new analytical techniques allows this masking effect to be quantified in terms of components-of-variance models. Moreover, we demonstrate how the components of variance associated with reader variability may be reduced when readers have the benefit of computer-assist reading aids. The remaining variability will be due to the case components, and these reflect the contribution of the technology without the masking effect of the reader. This suggests that prediction of clinical ranking of imaging systems in terms of physical measurements may become a much more tractable task in a world that includes MRMC ROC analysis of performance of radiologists with the advantage of computer-assisted reading.