Diagnostic capacity of skin tumor artificial intelligence-assisted decision-making software in real-world clinical settings

Background Youzhi artificial intelligence (AI) software is the AI-assisted decision-making system for diagnosing skin tumors. The high diagnostic accuracy of Youzhi AI software was previously validated in specific datasets. The objective of this study was to compare the performance of diagnostic capacity between Youzhi AI software and dermatologists in real-world clinical settings. Methods A total of 106 patients who underwent skin tumor resection in the Dermatology Department of China-Japan Friendship Hospital from July 2017 to June 2019 and were confirmed as skin tumors by pathological biopsy were selected. Dermoscopy and clinical images of 106 patients were diagnosed by Youzhi AI software and dermatologists at different dermoscopy diagnostic levels. The primary outcome was to compare the diagnostic accuracy of the Youzhi AI software with that of dermatologists and that measured in the laboratory using specific data sets. The secondary results included the sensitivity, specificity, positive predictive value, negative predictive value, F-measure, and Matthews correlation coefficient of Youzhi AI software in the real-world. Results The diagnostic accuracy of Youzhi AI software in real-world clinical settings was lower than that of the laboratory data (P < 0.001). The output result of Youzhi AI software has good stability after several tests. Youzhi AI software diagnosed benign and malignant diseases by recognizing dermoscopic images and diagnosed disease types with higher diagnostic accuracy than by recognizing clinical images (P = 0.008,P = 0.016, respectively). Compared with dermatologists, Youzhi AI software was more accurate in the diagnosis of skin tumor types through the recognition of dermoscopic images (P = 0.01). By evaluating the diagnostic performance of dermatologists under different modes, the diagnostic accuracy of dermatologists in diagnosing disease types by matching dermoscopic and clinical images was significantly higher than that by identifying dermoscopic and clinical images in random sequence (P = 0.022). The diagnostic accuracy of dermatologists in the diagnosis of benign and malignant diseases by recognizing dermoscopic images was significantly higher than that by recognizing clinical images (P = 0.010). Conclusion The diagnostic accuracy of Youzhi AI software for skin tumors in real-world clinical settings was not as high as that of using special data sets in the laboratory. However, there was no significant difference between the diagnostic capacity of Youzhi AI software and the average diagnostic capacity of dermatologists. It can provide assistant diagnostic decisions for dermatologists in the current state.