Development and Initial Testing of a Structured Clinical Observation Tool to Assess Pharmacotherapy Competence

Objective: The authors developed and tested the feasibility and utility of a new direct-observation instrument to assess trainee performance of a medication management session. Methods: The Psychopharmacotherapy-Structured Clinical Observation (P-SCO) instrument was developed based on multiple sources of expertise and then implemented in 4 university-based outpatient medication management clinics with 7 faculty supervising 17 third-year residents. After each observation by a faculty member of a medication management session, residents received feedback in writing (the completed P-SCO) and verbally in person. Targets were 8 P-SCO observations per academic year per resident (or 0.67 per month) and 16 observations per year completed by each faculty (or 1.3 per month). Qualitative thematic analysis was employed to compare the frequency, specificity, type (reinforcing vs. corrective), and content of comments documented on the P-SCO forms to mid-point and end of rotation global assessments by the same faculty for the same residents in the same rotation. Results: Faculty completed 2.4 (SD=1.2) P-SCOs per month during the study period. Each resident received 1.1 (SD=0.53) P-SCO observations per month. Faculty and residents completed significantly more observations than targeted (p=0.03 and p=0.003, respectively). Two percent of the P-SCOs had no written comments. Less than 3% of the P-SCO comments were nonspecific compared with 43% for the global assessments. Residents received, on average, 3.3 times more total, 2.6 times more reinforcing, and 5.3 times more corrective patient care specific comments on the P-SCO than on the global assessment (p<0.001). For the numerical ratings, residents received an average of 4.2 "exceeds expectations" and 1.7 "below expectations" ratings on P-SCOs compared with 2.6 and 0, respectively, on global assessments (p<0.02). Conclusion: Faculty can feasibly use the P-SCO instrument in a training clinic. Compared with traditional global assessment, the P-SCO provided much more specific feedback information, a better balance of corrective to re-enforcing comments, and a greater spread of ratings related to competency in pharmacotherapy.