Artificial intelligence for automatic and objective assessment of competencies in flexible bronchoscopy

Background: Bronchoscopy is a challenging technical procedure, and assessment of competence currently relies on expert raters. Human rating is time consuming and prone to rater bias. The aim of this study was to evaluate if a bronchial segment identification system based on artificial intelligence (AI) could automatically, instantly, and objectively assess competencies in flexible bronchoscopy in a valid way. Methods: Participants were recruited at the Clinical Skills Zone of the European Respiratory Society Annual Conference in Milan, 9th-13th September 2023. The participants performed one full diagnostic bronchoscopy in a simulated setting and were rated immediately by the AI according to its four outcome measures: diagnostic completeness (DC), structured progress (SP), procedure time (PT), and mean intersegmental time (MIT). The procedures were video-recorded and rated after the conference by two blinded, expert raters using a previously validated assessment tool with nine items regarding anatomy and dexterity. Results: Fifty-two participants from six different continents were included. All four outcome measures of the AI correlated significantly with the experts' anatomy-ratings (Pearson's correlation coefficient, P value): DC (r=0.47, P<0.001), SP (r=0.57, P<0.001), PT (r=-0.32, P=0.02), and MIT (r=-0.55, P<0.001) and also with the experts' dexterity-ratings: DC (r=0.38, P=0.006), SP (r=0.53, P<0.001), PT (r=-0.34, P=0.014), and MIT (r=-0.47, P<0.001). Conclusions: The study provides initial validity evidence for AI-based immediate and automatic assessment of anatomical and navigational competencies in flexible bronchoscopy. SP provided stronger correlations with human experts' ratings than the traditional DC.