Evaluating the Utility, Reliability, and Validity of a Resident Performance Evaluation Instrument

被引:7
作者
Tomisato, Shayne [1 ]
Venter, Jacob [2 ]
Weller, Jennifer [1 ]
Drachman, David [3 ]
机构
[1] Maricopa Integrated Hlth Syst, Dist Med Grp, Mesa, AZ 85210 USA
[2] Phoenix Childrens Hosp, Barrow Neurol Inst, Phoenix, AZ USA
[3] Maricopa Integrated Hlth Syst, Phoenix, AZ USA
关键词
Resident performance evaluation; Psychiatry Milestones; Psychiatry resident; Resident competence; COMPETENCE; EDUCATION; SKILLS; RISK;
D O I
10.1007/s40596-014-0134-7
中图分类号
G40 [教育学];
学科分类号
040101 ; 120403 ;
摘要
The primary goal of this project was to create and evaluate the utility, validity, and reliability of an instrument assessing resident performance during child and adolescent psychiatry (CAP) training. In three stages, the instrument was developed and evaluated for utility in assessing skills critical to CAP. Items on the Global Assessment Evaluation (GAE) were derived from the six core competencies identified by the American College of Graduate Medical Education (ACGME). Secondary goals included producing an instrument that could be used by other CAP training programs and describing a process that could be followed by training programs to create their own assessment instruments. Faculty members developed a CAP resident performance evaluation instrument. In a three-stage process, faculty utilized the tool to evaluate residents during and after rotations. Statistical findings guided revisions to improve the utility and reliability of the instrument. For the final version of the GAE, intra-class correlation coefficients were calculated to assess inter-rater reliability, and Principal Components Analysis provided further insight into the dimensions of resident assessment. The final version of the GAE showed overall and construct validity by capturing significant differences among residents and matched faculty members' overall impressions of resident performance. Intra-class correlation coefficient values for the overall score (0.945) and individual scales showed good reliability. Resident performance was not correlated with rotation site or model of care. The GAE has proved a valuable instrument in tracking the progress, strengths, and weaknesses in resident performance over the course of training. Data from multiple evaluations over time provide useful information about resident performance in a way that one or few evaluations does not. This finding is consistent with the practice of semiannual reviews and the new Clinical Competency Committees, both of which are required by the ACGME.
引用
收藏
页码:458 / 463
页数:6
相关论文
共 50 条
[41]   The PEDro scale had acceptably high convergent validity, construct validity, and interrater reliability in evaluating methodological quality of pharmaceutical trials [J].
Yamato, Tie Parma ;
Maher, Chris ;
Koes, Bart ;
Moseley, Anne .
JOURNAL OF CLINICAL EPIDEMIOLOGY, 2017, 86 :176-181
[42]   Development of an instrument for the evaluation of advanced life support performance [J].
Peltonen, L-M. ;
Peltonen, V. ;
Salantera, S. ;
Tommila, M. .
ACTA ANAESTHESIOLOGICA SCANDINAVICA, 2017, 61 (09) :1215-1231
[43]   Developing and evaluating the validity and reliability of the Iranian preschool food frequency questionnaire (IP-FFQ) [J].
Khorramrouz, Fatemeh ;
Khosravi, Maryam ;
Eslami, Kazem ;
Khadem-Rezaiyan, Majid .
CHILDREN AND YOUTH SERVICES REVIEW, 2024, 162
[44]   A neuropsychological instrument measuring age-related cerebral decline in older drivers: development reliability, and validity of MedDrive [J].
Vaucher, Paul ;
Cardoso, Isabel ;
Veldstra, Janet L. ;
Herzig, Daniela ;
Herzog, Michael ;
Mangin, Patrice ;
Favrat, Bernard .
FRONTIERS IN HUMAN NEUROSCIENCE, 2014, 8
[45]   Rasch Measurement Model for Validity and Reliability of Instrument Food Skills Competencies in addressing Food Neophobia for Preschool Teachers [J].
Saipudin, Nurul Ain ;
Suhairom, Nornazira ;
Abd Wahid, Nur Husna ;
Raof, Shuhada Abdul .
AKADEMIKA, 2023, 93 (03) :125-138
[46]   Improving handover competency in preclinical medical and health professions students: establishing the reliability and construct validity of an assessment instrument [J].
Michael, Meghan ;
Griggs, Andrew C. ;
Shields, Ian H. ;
Sadighi, Mozhdeh ;
Hernandez, Jessica ;
Chan, Chrissy ;
McHugh, Mary ;
Nichols, Blake E. ;
Joshi, Kavita ;
Testa, Daniel ;
Raj, Sonika ;
Preble, Richard ;
Lazzara, Elizabeth H. ;
Greilich, Philip E. .
BMC MEDICAL EDUCATION, 2021, 21 (01)
[47]   An Evaluation of an Intervention Programme in Teacher Training for Geography and History: A Reliability and Validity Analysis [J].
Rodriguez-Medina, Jairo ;
Gomez-Carrasco, Cosme J. ;
Miralles-Martinez, Pedro ;
Aznar-Diaz, Inmaculada .
SUSTAINABILITY, 2020, 12 (08)
[48]   Evaluation of the reliability and validity for X16 balance testing scale for the elderly [J].
Ju, Jingjuan ;
Jiang, Yu ;
Zhou, Peng ;
Li, Lin ;
Ye, Xiaolei ;
Wu, Hongmei ;
Shen, Bin ;
Zhang, Jialei ;
He, Xiaoding ;
Niu, Chunjin ;
Xia, Qinghua .
BMC GERIATRICS, 2018, 18
[49]   Turkish validity and reliability study of midwifery student evaluation of practice (MidSTEP) tool [J].
Aktas, Demet ;
Yilmaz, Sakine .
MIDWIFERY, 2024, 129
[50]   Validity and reliability of a protocol of orofacial myofunctional evaluation for patients with obstructive sleep apnea [J].
Folha, Gislaine A. ;
Valera, Fabiana C. P. ;
de Felicio, Claudia M. .
EUROPEAN JOURNAL OF ORAL SCIENCES, 2015, 123 (03) :165-172