Evaluation of an AI-based, automatic coronary artery calcium scoring software

Objectives To evaluate an artificial intelligence (AI)-based, automatic coronary artery calcium (CAC) scoring software, using a semi-automatic software as a reference. Methods This observational study included 315 consecutive, non-contrast-enhanced calcium scoring computed tomography (CSCT) scans. A semi-automatic and an automatic software obtained the Agatston score (AS), the volume score (VS), the mass score (MS), and the number of calcified coronary lesions. Semi-automatic and automatic analysis time were registered, including a manual double-check of the automatic results. Statistical analyses were Spearman's rank correlation coefficient (rho), intra-class correlation (ICC), Bland Altman plots, weighted kappa analysis (kappa), and Wilcoxon signed-rank test. Results The correlation and agreement for the AS, VS, and MS were rho = 0.935, 0.932, 0.934 (p < 0.001), and ICC = 0.996, 0.996, 0.991, respectively (p < 0.001). The correlation and agreement for the number of calcified lesions were rho = 0.903 and ICC = 0.977 (p < 0.001), respectively. The Bland Altman mean difference and 1.96 SD upper and lower limits of agreements for the AS, VS, and MS were - 8.2 (- 115.1 to 98.2), - 7.4 (- 93.9 to 79.1), and - 3.8 (- 33.6 to 25.9), respectively. Agreement in risk category assignment was 89.5% and kappa = 0.919 (p < 0.001). The median time for the semi-automatic and automatic method was 59 s (IQR 35-100) and 36 s (IQR 29-49), respectively (p < 0.001). Conclusions There was an excellent correlation and agreement between the automatic software and the semi-automatic software for three CAC scores and the number of calcified lesions. Risk category classification was accurate but showing an overestimation bias tendency. Also, the automatic method was less time-demanding.