Simulation-Based Design Optimization for Statistical Power: Utilizing Machine Learning

被引:3
作者
Zimmer, Felix [1 ]
Debelak, Rudolf [1 ]
机构
[1] Univ Zurich, Dept Psychol, Div Psychol Methods Evaluat & Stat, Binzmuehlestr 14,Box 27, CH-8050 Zurich, Switzerland
基金
瑞士国家科学基金会;
关键词
simulation; sample size; power analysis; machine learning; SAMPLE-SIZE; EFFICIENT; PACKAGE; TRIALS;
D O I
10.1037/met0000611
中图分类号
B84 [心理学];
学科分类号
04 ; 0402 ;
摘要
The planning of adequately powered research designs increasingly goes beyond determining a suitable sample size. More challenging scenarios demand simultaneous tuning of multiple design parameter dimensions and can only be addressed using Monte Carlo simulation if no analytical approach is available. In addition, cost considerations, for example, in terms of monetary costs, are a relevant target for optimization. In this context, optimal design parameters can imply a desired level of power at minimum cost or maximum power at a cost threshold. We introduce a surrogate modeling framework based on machine learning predictions to solve these optimization tasks. In a simulation study, we demonstrate the efficiency for a wide range of hypothesis testing scenarios with single- and multidimensional design parameters, including t tests, analysis of variance, item response theory models, multilevel models, and multiple imputations. Our framework provides an algorithmic solution for optimizing study designs when no analytic power analysis is available, handling multiple design dimensions and cost considerations. Our implementation is publicly available in the R package mlpwr.
引用
收藏
页数:25
相关论文
共 53 条
[1]   When power analyses based on pilot data are biased: Inaccurate effect size estimators and follow-up bias [J].
Albers, Casper ;
Lakens, Daniel .
JOURNAL OF EXPERIMENTAL SOCIAL PSYCHOLOGY, 2018, 74 :187-195
[2]   Statistical Power in Two-Level Models: A Tutorial Based on Monte Carlo Simulation [J].
Arend, Matthias G. ;
Schaefer, Thomas .
PSYCHOLOGICAL METHODS, 2019, 24 (01) :1-19
[3]   Power Contours: Optimising Sample Size and Precision in Experimental Psychology and Human Neuroscience [J].
Baker, Daniel H. ;
Vilidaite, Greta ;
Lygo, Freya A. ;
Smith, Anika K. ;
Flack, Tessa R. ;
Gouws, Andre D. ;
Andrews, Timothy J. .
PSYCHOLOGICAL METHODS, 2021, 26 (03) :295-314
[4]   Fitting Linear Mixed-Effects Models Using lme4 [J].
Bates, Douglas ;
Maechler, Martin ;
Bolker, Benjamin M. ;
Walker, Steven C. .
JOURNAL OF STATISTICAL SOFTWARE, 2015, 67 (01) :1-48
[5]   Advances in surrogate based modeling, feasibility analysis, and optimization: A review [J].
Bhosekar, Atharv ;
Ierapetritou, Marianthi .
COMPUTERS & CHEMICAL ENGINEERING, 2018, 108 :250-267
[6]  
Birnbaum A., 1968, Statistical theories of mental test scores, P395, DOI DOI 10.1002/J.2333-8504.1981.TB01255.X
[7]  
Bischl B, 2018, Arxiv, DOI [arXiv:1703.03373, DOI 10.48550/ARXIV.1703.03373, 10.48550/arXiv.1703.03373]
[8]   Initial Design Strategies and their Effects on Sequential Model-Based Optimization An Exploratory Case Study Based on BBOB [J].
Bossek, Jakob ;
Doerr, Carola ;
Kerschke, Pascal .
GECCO'20: PROCEEDINGS OF THE 2020 GENETIC AND EVOLUTIONARY COMPUTATION CONFERENCE, 2020, :778-786
[9]   Power failure: why small sample size undermines the reliability of neuroscience [J].
Button, Katherine S. ;
Ioannidis, John P. A. ;
Mokrysz, Claire ;
Nosek, Brian A. ;
Flint, Jonathan ;
Robinson, Emma S. J. ;
Munafo, Marcus R. .
NATURE REVIEWS NEUROSCIENCE, 2013, 14 (05) :365-376
[10]   A LIMITED MEMORY ALGORITHM FOR BOUND CONSTRAINED OPTIMIZATION [J].
BYRD, RH ;
LU, PH ;
NOCEDAL, J ;
ZHU, CY .
SIAM JOURNAL ON SCIENTIFIC COMPUTING, 1995, 16 (05) :1190-1208