Objective assessment of robotic surgical skill using instrument contact vibrations

Surgical skill evaluation ordinarily requires tedious video review and survey completion, while new automatic approaches focus on evaluating the quality of the surgeon's movements in free space. Robotic surgical instrument vibrations are simple to measure and physically correspond to how roughly instruments are handled, but they have yet to be studied as a measure of technical surgical skill. Thirteen surgeons used a robotic surgery system (da Vinci S by Intuitive Surgical) to perform four trials each of peg transfer (PT), needle pass (NP), and intracorporeal suturing (IS). Completion time, instrument vibrations, and applied forces were measured for each trial; root mean square (RMS) and total sum of squares (TSS) were calculated from both the vibration and force recordings. Four experienced surgeons blindly assessed the task videos using a Global Rating Scale (GRS), and skill metrics were compared between the eight novices and five experienced participants. Stepwise regression was performed to predict GRS score from objective skill metrics. The concurrent validity of each metric was evaluated using receiver operating characteristic (ROC) analysis. The GRS demonstrated excellent internal consistency (Cronbach's alpha = 0.91) and strong inter-rater reliability (ICC = 0.84). Compared to novices, experienced surgeons earned higher GRS scores and performed tasks with lower vibration magnitudes, lower forces, and shorter completion times in 15 of 18 task-metric combinations (p values ranging from 0.042 to < 0.001). ROC analysis demonstrated that including vibration and force magnitudes along with completion time in skill prediction models improves the objective classification of subjects as novice or experienced for all tasks studied (PT: 90 % sensitivity, 75 % specificity; NP: 85 % sensitivity, 84 % specificity; suturing: 100 % sensitivity, 100 % specificity). RMS and TSS instrument vibrations are novel construct-valid measures of robotic surgical skill that enable the development of objective skill assessment models comparable to observer-based ratings.