The reusable holdout: Preserving validity in adaptive data analysis

被引：189

作者：

Dwork, Cynthia ^{[1
]}

Feldman, Vitaly ^{[2
]}

Hardt, Moritz ^{[3
]}

Pitassi, Toniann ^{[4
]}

Reingold, Omer ^{[5
]}

Roth, Aaron ^{[6
]}

机构：

[1] Microsoft Res, Mountain View, CA 94043 USA

[2] IBM Almaden Res Ctr, San Jose, CA 95120 USA

[3] Google Res, Mountain View, CA 94043 USA

[4] Univ Toronto, Dept Comp Sci, Toronto, ON M5S 3G4, Canada

[5] Samsung Res Amer, Mountain View, CA 94043 USA

[6] Univ Penn, Dept Comp & Informat Sci, Philadelphia, PA 19104 USA

来源：

SCIENCE | 2015年 / 349卷 / 6248期

基金：

美国国家科学基金会; 加拿大自然科学与工程研究理事会;

关键词：

FALSE DISCOVERY; STABILITY;

D O I：

10.1126/science.aaa9375

中图分类号：

O [数理科学和化学]; P [天文学、地球科学]; Q [生物科学]; N [自然科学总论];

学科分类号：

07 ; 0710 ; 09 ;

摘要：

Misapplication of statistical data analysis is a common cause of spurious discoveries in scientific research. Existing approaches to ensuring the validity of inferences drawn from data assume a fixed procedure to be performed, selected before the data are examined. In common practice, however, data analysis is an intrinsically adaptive process, with new analyses generated on the basis of data exploration, as well as the results of previous analyses on the same data. We demonstrate a new approach for addressing the challenges of adaptivity based on insights from privacy-preserving data analysis. As an application, we show how to safely reuse a holdout data set many times to validate the results of adaptively chosen analyses.

引用

页码：636 / 638

页数：3

共 17 条

[1] The Quality Preserving Database: A Computational Framework for Encouraging Collaboration, Enhancing Power and Controlling False Discovery
Aharoni, Ehud
Neuvirth, Hani
Rosset, Saharon
[J]. IEEE-ACM TRANSACTIONS ON COMPUTATIONAL BIOLOGY AND BIOINFORMATICS, 2011, 8 (05) : 1431 - 1437
[2] CONTROLLING THE FALSE DISCOVERY RATE - A PRACTICAL AND POWERFUL APPROACH TO MULTIPLE TESTING
BENJAMINI, Y
HOCHBERG, Y
[J]. JOURNAL OF THE ROYAL STATISTICAL SOCIETY SERIES B-STATISTICAL METHODOLOGY, 1995, 57 (01) : 289 - 300
[3] Stability and generalization
Bousquet, O
Elisseeff, A
[J]. JOURNAL OF MACHINE LEARNING RESEARCH, 2002, 2 (03) : 499 - 526
[4] Cawley GC, 2010, J MACH LEARN RES, V11, P2079
[5] Chambers C., 2013, GUARDIAN US 0605
[6] Calibrating noise to sensitivity in private data analysis
Dwork, Cynthia
McSherry, Frank
Nissim, Kobbi
Smith, Adam
[J]. THEORY OF CRYPTOGRAPHY, PROCEEDINGS, 2006, 3876 : 265 - 284
[7] α-investing:: a procedure for sequential control of expected false discoveries
Foster, Dean P.
Stine, Robert A.
[J]. JOURNAL OF THE ROYAL STATISTICAL SOCIETY SERIES B-STATISTICAL METHODOLOGY, 2008, 70 : 429 - 444
[8] A NOTE ON SCREENING REGRESSION EQUATIONS
FREEDMAN, DA
[J]. AMERICAN STATISTICIAN, 1983, 37 (02) : 152 - 155
[9] The Statistical Crisis in Science
Gelman, Andrew
Loken, Eric
[J]. AMERICAN SCIENTIST, 2014, 102 (06) : 460 - 465
[10] Hastie T., 2009, The elements of statistical learning: data mining, inference, and prediction, V2nd

← 1 2 →