Evaluating resident operative performance: A qualitative analysis of expert opinions

Purpose. To elicit and analyze the strengths and weaknesses of resident operative performance (ROP) as identified by independent expert ratings. Methods. Four groups of expert surgeons, blinded to resident training level, evaluated ROP on 2 each of laparoscopic cholecystectomy, thyroidectomy, open inguinal, and laparoscopic ventral hernia repair audio videos, and listed strengths and weaknesses. Comments were coded as technical skills, forward planning, self-direction, situation awareness, and patient safety, and as either procedure specific or general skills. Results. Eighteen experts independently entered 1087 comments on 8 ROP strengths and weakness. In e post independent rating discussion, consensus was reached on 85 (28%) of 300 post rating comments with majority agreement on another 83 (28%). Overall, the dominant focus was on forward planning. With the exception of the laparoscopic cholecystectomy cases, raters focused more on general than on procedure-specific skills (P < .05). Conclusion. Fewer than 30% of expert rater comments focused on technical skills when considering ROP strengths and weaknesses. Although there was some variation in individual comments, majority agreement was reached on 56% of comments during the post independent rating discussion. These findings will improve rater training and further the implementation of a national assessment process to evaluate end of training surgical competence and operative proficiency. (Surgery 2011;150:759-70.)