Effect of Choice of Estimation Method on Inter-Hospital Mortality Rate Comparisons

Objective: To evaluate and compare the use of 6 different methods for calculating expected mortality rates and standardized mortality ratios (SMRs) when performing interhospital mortality rate comparisons. Design: Retrospective cohort study using actual and simulated hospitalization data to evaluate the use of (1) fixed-effects, (2) Generalized Linear Mixed Model, and (3) Bayesian (Markov Chain Monte Carlo--based) random-effects models on both aggregated and individual-level data to estimate SMRs by hospital. Setting: Seventeen hospitals in a large integrated health care delivery system. Main Outcome Measure: Inpatient mortality. Results: Results from the 6 different methods compared in this study were highly correlated both on log(SMR) values and hospital ranks (range, 0.91-1). All the methods had high specificity (> 87%) for finding true underlying mortality effects. The fixed-effects models had higher overall sensitivity than the random-effects models. The individual-level random effects model had generally higher sensitivity than the aggregated random-effects models. All methods showed a high correlation with the true ranks. Discussion: When comparing mortality rates across hospitals, it is important to focus not only on the method used to measure patient sickness but also on the analytical technique used to estimate hospital-specific adjusted mortality rates. When an illness severity measure including detailed physiologic data was used, the simplest method we examined, a fixed-effects aggregate-level approach in common usage, out-performed the other methods when both specificity and sensitivity are considered. Use of a severity measure that correlates less well with mortality than the one we employed would be expected to reduce the sensitivity of all of the methods examined.