Evaluating health risk models

Interest in targeted disease prevention has stimulated development of models that assign risks to individuals, using their personal covariates. We need to evaluate these models and quantify the gains achieved by expanding a model to include additional covariates. This paper reviews several performance measures and shows how they are related. Examples are used to show that appropriate performance criteria for a risk model depend upon how the model is used. Application of the performance measures to risk models for hypothetical populations and for US women at risk of breast cancer illustrate two additional points. First, model performance is constrained by the distribution of risk-determining covariates in the population. This complicates the comparison of two models when applied to populations with different covariate distributions. Second, all summary performance measures obscure model features of relevance to its utility for the application at hand, such as performance in specific subgroups of the population. In particular, the precision gained by adding covariates to a model can be small overall, but large in certain subgroups. We propose new ways to identify these subgroups and to quantify how much they gain by measuring the additional covariates. Those with largest gains could be targeted for cost-efficient covariate assessment. Copyright (C) 2010 John Wiley & Sons, Ltd.