Reliability of utilization measures for primary care physician profiling

Background: Given rising health care costs, there has been a renewed interest in using utilization measures to profile physicians. Despite the measures' common use, few studies have examined their reliability and whether they capture true differences among physicians. Methods: A local health improvement organization in New York State used 2008-2010 claims data to create 11 utilization measures for feedback to primary care physicians (PCP). The sample consists of 2938 PCPs in 1546 practices who serve 853,187 patients. We used these data to measure reliability of these utilization measures using two methods (hierarchical model versus test-retest). For each PCP and each practice, we estimate each utilization measure's reliability, ranging from 0 to 1, with 0 indicating that all differences in utilization are due to random noise and 1 indicating that all differences are due to real variation among physicians. Results: Reliability varies significantly across the measures. For 4 utilization measures (PCP visits, specialty visits, PCP lab tests (blood and urine), and PCP radiology and other tests), reliability was high (mean > 0.85) at both the physician and the practice level. For the other 7 measures (professional therapeutic visits, emergency room visits, hospital admissions, readmissions, skilled nursing facility days, skilled home care visits, and custodial home care services), there was lower reliability indicating more substantial measurement error. Conclusions: The results illustrate that some utilization measures are suitable for PCP and practice profiling while caution should be used when using other utilization measures for efforts such as public reporting or pay-for-performance incentives.(C) 2013 Elsevier Inc. All rights reserved.