Development and evaluation of a virtual knowledge assessment tool for transanal total mesorectal excision

Background Transanal total mesorectal excision (TATME) is difficult to learn and can result in serious complications. Current paradigms for assessing performance and competency may be insufficient. This study aims to develop and provide preliminary validity evidence for a TATME virtual assessment tool (TATME-VAT) to assess the cognitive skills necessary to safely complete TATME dissection. Methods Participants from North America, Europe, Japan and China completed the test via an interactive online platform between 11/2019 and 05/2020. They were grouped into expert, experienced and novice surgeons depending on the number of independently performed TATMEs. TATME-VAT is a 24-item web-based assessment evaluating advanced cognitive skills, designed according to a blueprint from consensus guidelines. Eight items were multiple choice questions. Sixteen items required making annotations on still frames of TATME videos (VCT) and were scored using a validated algorithm derived from experts' responses. Annotation (range 0-100), multiple choice (range 0-100), and overall scores (sum of annotation and multiple-choice scores, normalized to mu = 50 and sigma = 10) were reported. Results There were significant differences between the expert, experienced, and novice groups for the annotation (p <0.001), multiple-choice (p < 0.001), and overall scores (p < 0.001). The annotation (p = 0.439) and overall (p = 0.152) scores were similar between the experienced and novice groups. Annotation scores were higher in participants with 51 or more vs. 30-50 vs. less than 30 cases. Scores were also lower in users with a self-reported recent complication vs. those without. Conclusions This study describes the development of an interactive video-based virtual assessment tool for TATME dissection and provides initial validity evidence for its use.