Investigation of standardized patient ratings of humanistic competence on a medical licensure examination using Many-Facet Rasch Measurement and generalizability theory

Humanistic doctor–patient interaction has been measured for eight years using the Global Patient Assessment (GPA) tool in the national osteopathic clinical skills medical licensure examination. Standardized patients (SPs) apply the GPA tool to rate examinees’ competence on doctor–patient communication, interpersonal skills, and professionalism. Many-Facet Rasch Measurement was employed to evaluate the overall functioning of the GPA rating scale and to estimate measurement errors associated with characteristics of SP raters and medical case presentations. Generalizability theory was applied to investigate variance components corresponding to each facet of interest. For the 2010–2011 testing cycle, 50,090 SP ratings were analyzed. Although SP raters varied in leniency/stringency of rating, SPs differentiated the six GPA aspects in difficulty, and utilized a reasonable range of the 9-point scale. Reliability indices resulted in sufficient examinee separation, 0.94, from the Rasch model and sufficient dependability from the generalizability analysis for raw scores, 0.83, and transformed Rasch scores, 0.97. Results indicate that medical students’ humanistic competence can be reliably measured through the GPA tool in the simulated environment. These measurement models supplement other means of observation and quality control with valuable information about the psychometric quality of SP ratings of humanistic competence.