Simulation-based assessment in anesthesia: an international multicentre validation study

被引:0
作者
Everett, Tobias [1 ]
Mackinnon, Ralph [2 ]
Ng, Elaine [1 ]
Kulkarni, Pradeep [3 ]
Borges, Bruno C. R. [4 ]
Letal, Michael [5 ]
Fleming, Melinda [6 ]
Bould, M. Dylan [7 ]
Marsh, Christopher [8 ]
Heather, David [9 ]
Colovic, Vesna [2 ]
Kulcsar, Zsuzsanna [2 ]
Boyle, Riley [10 ]
机构
[1] Univ Toronto, Hosp Sick Children, Dept Anesthesia & Pain Med, 555 Univ Ave, University, ON M5G 1X8, Canada
[2] Royal Manchester Childrens Hosp, Dept Anesthesia, Manchester, Lancs, England
[3] Univ Alberta, Stollery Childrens Hosp, Dept Anesthesia, Edmonton, AB, Canada
[4] McMaster Univ, McMaster Childrens Hosp, Dept Anesthesia, Hamilton, ON, Canada
[5] Univ Calgary, Alberta Childrens Hosp, Dept Anesthesia, Calgary, AB, Canada
[6] Queens Univ, Dept Anesthesia, Kingston, ON, Canada
[7] Univ Ottawa, Childrens Hosp Eastern Ontario, Dept Anesthesia, Ottawa, ON, Canada
[8] Royal United Hosp, Dept Anesthesia, Bath, Somerset, England
[9] Middlemore Hosp, Dept Anesthesia, Auckland, New Zealand
[10] Stollery Childrens Hosp, Dept Anesthesia, Edmonton, AB, Canada
来源
CANADIAN JOURNAL OF ANESTHESIA-JOURNAL CANADIEN D ANESTHESIE | 2019年 / 66卷 / 12期
关键词
VALIDITY EVIDENCE; PERFORMANCE; RESIDENTS; AGREEMENT; SKILLS;
D O I
10.1007/s12630-019-01488-4
中图分类号
R614 [麻醉学];
学科分类号
100217 ;
摘要
Purpose Simulated clinical events provide a means to evaluate a practitioner's performance in a standardized manner for all candidates that are tested. We sought to provide evidence for the validity of simulation-based assessment tools in simulated pediatric anesthesia emergencies. Methods Nine centres in two countries recruited subjects to participate in simulated operating room events. Participants ranged in anesthesia experience from junior residents to staff anesthesiologists. Performances were video recorded for review and scored by specially trained, blinded, expert raters. The rating tools consisted of scenario-specific checklists and a global rating scale that allowed the rater to make a judgement about the subject's performance, and by extension, preparedness for independent practice. The reliability of the tools was classified as "substantial" (intraclass correlation coefficients ranged from 0.84 to 0.96 for the checklists and from 0.85 to 0.94 for the global rating scale). Results Three-hundred and ninety-one simulation encounters were analysed. Senior trainees and staff significantly out-performed junior trainees (P = 0.04 and P < 0.001 respectively). The effect size of grade (junior vs senior trainee vs staff) on performance was classified as "medium" (partial eta(2) = 0.06). Performance deficits were observed across all grades of anesthesiologist, particularly in two of the scenarios. Conclusions This study supports the validity of our simulation-based anesthesiologist assessment tools in several domains of validity. We also describe some residual challenges regarding the validity of our tools, some notes of caution in terms of the intended consequences of their use, and identify opportunities for further research.
引用
收藏
页码:1440 / 1449
页数:10
相关论文
共 26 条
  • [1] Incorporating simulation-based objective structured clinical examination into the Israeli national board examination in anesthesiology
    Berkenstadt, H
    Ziv, A
    Gafni, N
    Sidi, A
    [J]. ANESTHESIA AND ANALGESIA, 2006, 102 (03) : 853 - 858
  • [2] Simulation-based Assessment to Identify Critical Gaps in Safe Anesthesia Resident Performance
    Blum, Richard H.
    Boulet, John R.
    Cooper, Jeffrey B.
    Muret-Wagstaff, Sharon L.
    [J]. ANESTHESIOLOGY, 2014, 120 (01) : 129 - 141
  • [3] Linking Simulation-Based Educational Assessments and Patient-Related Outcomes: A Systematic Review and Meta-Analysis
    Brydges, Ryan
    Hatala, Rose
    Zendejas, Benjamin
    Erwin, Patricia J.
    Cook, David A.
    [J]. ACADEMIC MEDICINE, 2015, 90 (02) : 246 - 256
  • [4] Chiu M, 2016, CAN J ANESTH, V63, P1357, DOI 10.1007/s12630-016-0733-8
  • [5] Cohen J, 1988, Statistical Power Analysis for the Behavioral Sciences, VSecond
  • [6] What counts as validity evidence? Examples and prevalence in a systematic review of simulation-based assessment
    Cook, David A.
    Zendejas, Benjamin
    Hamstra, Stanley J.
    Hatala, Rose
    Brydges, Ryan
    [J]. ADVANCES IN HEALTH SCIENCES EDUCATION, 2014, 19 (02) : 233 - 250
  • [7] Technology-Enhanced Simulation to Assess Health Professionals: A Systematic Review of Validity Evidence, Research Methods, and Reporting Quality
    Cook, David A.
    Brydges, Ryan
    Zendejas, Benjamin
    Hamstra, Stanley J.
    Hatala, Rose
    [J]. ACADEMIC MEDICINE, 2013, 88 (06) : 872 - 883
  • [8] Culley D, 2015, ANESTHESIOLOGY MILES
  • [9] Ten years of simulation-based training in pediatric anesthesia: The inception, evolution, and dissemination of the Managing Emergencies in Pediatric Anesthesia (MEPA) course
    Everett, Tobias C.
    MacKinnon, Ralph
    de Beer, David
    Taylor, Matthew
    Bould, Matthew Dylan
    [J]. PEDIATRIC ANESTHESIA, 2017, 27 (10) : 984 - 990
  • [10] The Managing Emergencies in Paediatric Anaesthesia global rating scale is a reliable tool for simulation-based assessment in pediatric anesthesia crisis management
    Everett, Tobias C.
    Elaine Ng
    Power, Daniel
    Marsh, Christopher
    Tolchard, Stephen
    Shadrina, Anna
    Bould, Matthew D.
    [J]. PEDIATRIC ANESTHESIA, 2013, 23 (12) : 1117 - 1123