Reliability adjustment: A necessity for trauma center ranking and benchmarking

BACKGROUND: Currently, trauma center quality benchmarking is based on risk adjusted observed-expected (O/E) mortality ratios. However, failure to account for number of patients has been recently shown to produce unreliable mortality estimates, especially for low-volume centers. This study explores the effect of reliability adjustment (RA), a statistical technique developed to eliminate bias introduced by low volume on risk-adjusted trauma center benchmarking. METHODS: Analysis of the National Trauma Data Bank 2010 was performed. Patients 16 years or older with blunt or penetrating trauma and an Injury Severity Score (ISS) of 9 or greater were included. Based on the statistically accepted standards of the Trauma Quality Improvement Program methodology, risk-adjusted mortality rates were generated for each center and used to rank them accordingly. Hierarchical logistic regression modeling was then performed to adjust these rates for reliability using an empiric Bayes approach. The impact of RA was examined by (1) recalculating interfacility variations in adjusted mortality rates and (2) comparing adjusted hospital mortality quintile rankings before and after RA. RESULTS: A total of 557 facilities (with 278,558 patients) were included. RA significantly reduced the variation in risk-adjusted mortality rates between centers from 14-fold (0.7-9.8%) to only 2-fold (4.4-9.6%) after RA. This reduction in variation was most profound for smaller centers. A total of 68 "best'' hospitals and 18 "worst'' hospitals based on current risk adjustment methods were reclassified after performing RA. CONCLUSION: "Reliability adjustment'' dramatically reduces variations in risk-adjusted mortality arising from statistical noise, especially for lower volume centers. Moreover, the absence of RA had a profound impact on hospital performance assessment, suggesting that nearly one of every six hospitals in National Trauma Data Bank would have been inappropriately placed among the very best or very worst quintile of rankings. RA should be considered while benchmarking trauma centers based on mortality. (J Trauma Acute Care Surg. 2013; 75: 166-172. Copyright (C) 2013 by Lippincott Williams & Wilkins)