Determining relative importance of variables in developing and validating predictive models

被引:39
作者
Beyene, Joseph [1 ,2 ,3 ]
Atenafu, Eshetu G.
Hamid, Jemila S.
To, Teresa [2 ,3 ]
Sung, Lillian [1 ,3 ]
机构
[1] Hosp Sick Children, Div Hematol Oncol, Toronto, ON M5G 1X8, Canada
[2] Univ Toronto, Dalla Lana Sch Publ Hlth, Toronto, ON, Canada
[3] Univ Toronto, Dept Hlth Policy Management & Evaluat, Toronto, ON, Canada
来源
BMC MEDICAL RESEARCH METHODOLOGY | 2009年 / 9卷
关键词
PROSTATE-SPECIFIC ANTIGEN; TUMOR LYSIS SYNDROME; MEDICAL JOURNALS; REGRESSION; SELECTION; CANCER; FEATURES; OUTCOMES; RISK;
D O I
10.1186/1471-2288-9-64
中图分类号
R19 [保健组织与事业(卫生事业管理)];
学科分类号
摘要
Background: Multiple regression models are used in a wide range of scientific disciplines and automated model selection procedures are frequently used to identify independent predictors. However, determination of relative importance of potential predictors and validating the fitted models for their stability, predictive accuracy and generalizability are often overlooked or not done thoroughly. Methods: Using a case study aimed at predicting children with acute lymphoblastic leukemia (ALL) who are at low risk of Tumor Lysis Syndrome (TLS), we propose and compare two strategies, bootstrapping and random split of data, for ordering potential predictors according to their relative importance with respect to model stability and generalizability. We also propose an approach based on relative increase in percentage of explained variation and area under the Receiver Operating Characteristic (ROC) curve for developing models where variables from our ordered list enter the model according to their importance. An additional data set aimed at identifying predictors of prostate cancer penetration is also used for illustrative purposes. Results: Age is chosen to be the most important predictor of TLS. It is selected 100% of the time using the bootstrapping approach. Using the random split method, it is selected 99% of the time in the training data and is significant (at 5% level) 98% of the time in the validation data set. This indicates that age is a stable predictor of TLS with good generalizability. The second most important variable is white blood cell count (WBC). Our methods also identified an important predictor of TLS that was otherwise omitted if relying on any of the automated model selection procedures alone. A group at low risk of TLS consists of children younger than 10 years of age, without T-cell immunophenotype, whose baseline WBC is < 20 x 10(9)/L and palpable spleen is < 2 cm. For the prostate cancer data set, the Gleason score and digital rectal exam are identified to be the most important indicators of whether tumor has penetrated the prostate capsule. Conclusion: Our model selection procedures based on bootstrap re-sampling and repeated random split techniques can be used to assess the strength of evidence that a variable is truly an independent and reproducible predictor. Our methods, therefore, can be used for developing stable and reproducible models with good performances. Moreover, our methods can serve as a good tool for validating a predictive model. Previous biological and clinical studies support the findings based on our selection and validation strategies. However, extensive simulations may be required to assess the performance of our methods under different scenarios as well as check their sensitivity to a random fluctuation in the data.
引用
收藏
页数:10
相关论文
共 36 条
[1]   STATISTICS IN MEDICAL JOURNALS - DEVELOPMENTS IN THE 1980S [J].
ALTMAN, DG .
STATISTICS IN MEDICINE, 1991, 10 (12) :1897-1913
[2]  
Altman DG, 2000, STAT MED, V19, P453, DOI 10.1002/(SICI)1097-0258(20000229)19:4<453::AID-SIM350>3.3.CO
[3]  
2-X
[4]   How well does the gleason score predict prostate cancer death?: A 20-year followup of a population based cohort in Sweden [J].
Andrén, O ;
Fall, K ;
Franzén, L ;
Andersson, SO ;
Johansson, JE ;
Rubin, MA .
JOURNAL OF UROLOGY, 2006, 175 (04) :1337-1340
[5]   Automated variable selection methods for logistic regression produced unstable models for predicting acute myocardial infarction mortality [J].
Austin, PC ;
Tu, JV .
JOURNAL OF CLINICAL EPIDEMIOLOGY, 2004, 57 (11) :1138-1146
[6]   Bootstrap methods for developing predictive models [J].
Austin, PC ;
Tu, JV .
AMERICAN STATISTICIAN, 2004, 58 (02) :131-137
[7]   A Prediction Model for Lung Cancer Diagnosis that Integrates Genomic and Clinical Features [J].
Beane, Jennifer ;
Sebastiani, Paola ;
Whitfield, Theodore H. ;
Steiling, Katrina ;
Dumas, Yves-Martine ;
Lenburg, Marc E. ;
Spira, Avrum .
CANCER PREVENTION RESEARCH, 2008, 1 (01) :56-64
[8]  
Bender R, 1996, BMJ-BRIT MED J, V313, P628
[9]   Statistical modeling: The two cultures [J].
Breiman, L .
STATISTICAL SCIENCE, 2001, 16 (03) :199-215
[10]  
Carpenter J, 2000, STAT MED, V19, P1141, DOI 10.1002/(SICI)1097-0258(20000515)19:9<1141::AID-SIM479>3.0.CO