Forming a reference standard from LIDC data: Impact of reader agreement on reported CAD performance

The Lung Image Database Consortium (LIDC) has provided a publicly available collection of CT images with nodule markings from four radiologists. The LIDC protocol does not require radiologists to reach a consensus during the reading process, and as a result, there are varying levels of reader agreement for each potential nodule with no explicit reference standard for nodules. The purpose of this work was to investigate the effects of the level of reader agreement on the development of a reference standard and the subsequent impact on CAD performance. Ninety series were downloaded from the LIDC database. Four different reference standards were created based on the markings of the LIDC radiologists, reflecting four different levels of reader agreement. All series were analyzed with a research CAD system and its performance was measured against each of the four standards. Between the standards with the lowest (any I of 4 readers) and highest (all 4 readers) required level of reader agreement, the number of nodules ! 3 mm decreased 4.8% (from 174 to 90) and CAD sensitivity for nodules >= 3 mm increased from 0.70 +/- 0.34 to 0.79 +/- 0.35. Between the same reference standards, the number of nodules < 3 mm decreased 84% (from 483 to 75) and CAD sensitivity for nodules < 3 mm increased from 0.30 +/- 0.29 to 0.51 +/- 0.45. This research illustrates the importance of indicating the method used to form the reference standard, since the method influences both the number of nodules and reported CAD performance.