Direct Observation Evaluations by Emergency Medicine Faculty Do Not Provide Data That Enhance Resident Assessment When Compared to Summative Quarterly Evaluations

被引:0
作者
Ryan, James G. [1 ]
Barlas, David [1 ]
Sharma, Manish [1 ]
机构
[1] New York Hosp Queens, Dept Emergency Med, Flushing, NY USA
关键词
residency; evaluation; observation; OBSERVATION ASSESSMENT-TOOL; RELIABILITY; PERFORMANCE; COMPETENCE; TIME;
D O I
10.1111/j.1553-2712.2010.00878.x
中图分类号
R4 [临床医学];
学科分类号
1002 ; 100602 ;
摘要
Objectives: The purpose of this study was to compare quarterly global evaluations with direct observation evaluations to determine if direct observation evaluations provide unique data compared to those obtained from quarterly global evaluations. Methods: This observational, cohort study was performed at a 3-year emergency medicine (EM) residency program with 10 residents per year. Faculty used an online Web-based evaluation system to complete quarterly global evaluations and patient-specific direct observation evaluations. Two scores were collected for each resident within each quarterly evaluation period: 1) the quarterly evaluation score was the mean score across all faculty who performed a quarterly evaluation and, 2) the direct observation score was the mean score across all faculty who performed a direct observation evaluation. Pearson correlation coefficients were performed across these two groups of evaluations. Results: Over the 4-year period of the study 296 complete data sets were available for the analysis. When the quarterly evaluation score was correlated with the direct observation score for each resident at the same evaluation period, we found a very high correlation for each of the eight evaluation questions (r = 0.95-0.96, p < 0.0001). When these evaluations were stratified based on the number of direct observation evaluations that were performed during the evaluation period of interest, the correlation between the quarterly evaluation and the direct observation scores increased as the number of direct observations in the evaluation period increased. The evaluation scores from the faculty who had performed both direct observation and quarterly evaluation methods during the same resident evaluation period were highly correlated even with small numbers of evaluators. Conclusions: Direct observations are highly correlated with quarterly evaluations when there are greater than three direct observation evaluations completed; however, this correlation drops significantly when the number of direct observations is lower. Direct observation evaluations provide similar data when compared with data obtained from quarterly global evaluations. ACADEMIC EMERGENCY MEDICINE 2010; 17:S72-S77 (C) 2010 by the Society for Academic Emergency Medicine.
引用
收藏
页码:S72 / S77
页数:6
相关论文
共 10 条
[1]   Reliability of performance-based clinical skill assessment of emergency medicine residents [J].
Burdick, WP ;
BenDavid, MF ;
Swisher, L ;
Becher, J ;
Magee, D ;
McNamara, R ;
Zwanger, M .
ACADEMIC EMERGENCY MEDICINE, 1996, 3 (12) :1119-1123
[2]   An evaluation of emergency medicine resident interaction time with faculty in different teaching venues [J].
Chisholm, CD ;
Whenmouth, LF ;
Daly, EA ;
Cordell, WH ;
Giles, BK ;
Brizendine, EJ .
ACADEMIC EMERGENCY MEDICINE, 2004, 11 (02) :149-155
[3]   Evaluation of resident performance and intensive bedside teaching during direct observation [J].
Cydulka, RK ;
Emerman, CL ;
Jouriles, NJ .
ACADEMIC EMERGENCY MEDICINE, 1996, 3 (04) :345-351
[4]   Clinical assessment in emergency medicine [J].
Jouriles, N ;
Burdick, W ;
Hobgood, C .
ACADEMIC EMERGENCY MEDICINE, 2002, 9 (11) :1289-1294
[5]   Patient care competency in emergency medicine graduate medical education: Results of a consensus group on patient care [J].
King, RW ;
Schiavone, F ;
Counselman, FL ;
Panacek, EA .
ACADEMIC EMERGENCY MEDICINE, 2002, 9 (11) :1227-1235
[6]   Real-Time Inter-Rater Reliability of the Council of Emergency Medicine Residency Directors Standardized Direct Observation Assessment Tool [J].
LaMantia, Joseph ;
Kane, Bryan ;
Yarris, Lalena ;
Tadros, Anthony ;
Ward, Mary Frances ;
Lesser, Martin ;
Shayne, Philip .
ACADEMIC EMERGENCY MEDICINE, 2009, 16 (12) :S51-S57
[7]   Reliability of faculty clinical evaluations of non-emergency medicine residents during emergency department rotations [J].
Ryan, JG ;
Mandel, FS ;
Sama, A ;
Ward, MF .
ACADEMIC EMERGENCY MEDICINE, 1996, 3 (12) :1124-1130
[8]   Reliability of a core competency checklist assessment in the emergency department: The standardized direct observation assessment tool [J].
Shayne, Philip ;
Gallahue, Fiona ;
Rinnert, Stephan ;
Anderson, Craig L. ;
Hern, Gene ;
Katz, Eric .
ACADEMIC EMERGENCY MEDICINE, 2006, 13 (07) :727-732
[9]   Feasibility, reliability and user satisfaction with a PDA-Based Mini-CEX to evaluate the clinical skills of third-year medical students [J].
Torre, Dario M. ;
Simpson, Deborah E. ;
Elnicki, D. Michael ;
Sebastian, James L. ;
Holmboe, Eric S. .
TEACHING AND LEARNING IN MEDICINE, 2007, 19 (03) :271-277
[10]   Assuring the reliability of resident performance appraisals: More items or more observations? [J].
Williams, RG ;
Verhulst, S ;
Colliver, JA ;
Dunnington, GL .
SURGERY, 2005, 137 (02) :141-147