Impact of Artificial Intelligence- Assisted Indication Selection on Appropriateness Order Scoring for Imaging Clinical Decision Support

Purpose: The aim of this study was to assess appropriateness scoring and structured order entry after the implementation of an artificial intelligence (AI) tool for analysis of free-text indications. Methods: Advanced outpatient imaging orders in a multicenter health care system were recorded 7 months before (March 1, 2020, to September 21, 2020) and after (October 20, 2020, to May 13, 2021) the implementation of an AI tool targeting free-text indications. Clinical decision support score (not appropriate, may be appropriate, appropriate, or unscored) and indication type (structured, free-text, both, or none) were assessed. The X-2 and multivariate logistic regression adjusting for covariables with bootstrapping were used.Results: In total, 115,079 orders before and 150,950 orders after AI tool deployment were analyzed. The mean patient age was 59.3 +/- 15.5 years, and 146,035 (54.9%) were women; 49.9% of orders were for CT, 38.8% for MR, 5.9% for nuclear medicine, and 5.4% for PET. After deployment, scored orders increased to 52% from 30% (P < .001). Orders with structured indications increased to 67.3% from 34.6% (P < .001). On multivariate analysis, orders were more likely to be scored after tool deployment (odds ratio [OR], 2.7, 95% CI, 2.63-2.78; P < .001). Compared with physicians, orders placed by nonphysician providers were less likely to be scored (OR, 0.80; 95% CI, 0.78-0.83; P < .001). MR (OR, 0.84; 95% CI, 0.82-0.87) and PET (OR, 0.12; 95% CI, 0.10-0.13) were less likely to be scored than CT (; P < .001). After AI tool deployment, 72,083 orders (47.8%) remained unscored, 45,186 (62.7%) with free-text-only indications. Conclusions: Embedding AI assistance within imaging clinical decision support was associated with increased structured indication orders and independently predicted a higher likelihood of scored orders. However, 48% of orders remained unscored, driven by both provider behavior and infrastructure-related barriers.