Inter-observer consistency on subsolid nodule follow-up recommendation based on National Comprehensive Cancer Network (NCCN) guidelines in low-dose computed tomography (LDCT) lung cancer screening

Background: Follow-up management of pulmonary nodules is a crucial component of lung cancer screening. Consistency in follow-up recommendations is essential for effective lung cancer screening. This study aimed to assess inter-observer agreement on National Comprehensive Cancer Network (NCCN) guideline-based follow-up recommendation for subsolid nodules from low-dose computed tomography (LDCT) screening. Methods: A retrospective collection of LDCT reports from 2014 to 2017 for lung cancer screening was conducted using the Radiology Information System and keyword searches, focusing on subsolid nodules. A total of 110 LDCT cases containing subsolid nodules were identified. Two senior radiologists provided standardized follow-up recommendation. Follow-up recommendation was categorized into four groups (0-, 3-, 6-, and 12-month). To ensure overall balance and representativeness of the follow-up categories, 60 scans from 60 participants were included (distribution ratio 1:1:2:2). Cases were categorised into follow-up recommendation groups by five observers following NCCN guidelines. Fleiss' kappa statistic was used to evaluate inter-observer agreement. Results: Overall accuracy rate for follow-up recommendation among five observers was 72.3%. Chest radiologists' overall agreement was significantly higher than radiology residents (P<0.01). The overall agreement among the five observers was moderate, with a Fleiss' kappa of 0.437. For all paired readers, the mean Cohen's kappa value was 0.603, with 95% confidence interval (CI) from 0.489 to 0.716. Chest radiologists demonstrated substantial agreement, evidenced by a Cohen's kappa of 0.655 (95% CI: 0.503- 0.807). In contrast, the mean Cohen's kappa among radiology residents was 0.533 (95% CI: 0.501-0.565). The majority of cases with discrepancies, accounting for 73.5%, were associated with the same risk-dominant nodules. A higher proportion of part-solid nodule was a risk factor for discrepancies. Of the 600 paired readings, major discrepancies and substantial discrepancies were observed in 27.5% and 4.8% (29/600) of the cases. Conclusions: In subsolid nodules, category evaluation of observer follow-up recommendation based on NCCN guidelines achieved moderate consistency. Disagreements were mainly caused by measurement and type disagreements of identical risk-dominant nodules. Part-solid nodule was a contributor for discrepancies in follow-up recommendation. Major and substantial management discrepancies were 27.5% and 4.8% in the paired evaluations.