Generalizability of Statistical Prediction From Psychological Assessment Data: An Investigation With the MMPI-2-RF

In the present study, the author employed tools and principles from the domain of machine learning to investigate four questions related to the generalizability of statistical prediction in psychological assessment. First, to what extent do predictive methods common to psychology research and machine learning actually tend to predict new data points in new settings? Second, of what practical value is parsimony in applied prediction? Third, what is the most effective way to select model predictors when attempting to maximize generalizability? Fourth, how well do the methods considered compare with one another with respect to prediction generalizability? To address these questions, the author developed various types of predictive models on the basis of Minnesota Multiphasic Personality Inventory (MMPI)-2-RF scales, using multiple prediction criteria, in a calibration inpatient sample, then externally validated those models by applying them to one or two clinical samples from other settings. Model generalizability was then evaluated based on prediction accuracy in the external validation samples. Noteworthy findings from the present study include (a) statistical models generally demonstrated observable performance shrinkage across settings regardless of modeling approach, though they nevertheless tended to retain non-negligible predictive power in new settings; (b) of the modeling approaches considered, regularized (penalized) regression methods appeared to produce the most consistently robust predictions across settings; (c) parsimony appeared more likely to reduce than to enhance model generalizability; and (d) multivariate models whose predictors were selected automatically tended to perform relatively well, often producing substantially more generalizable predictions than models whose predictors were selected based on theory.