MALL language learning outcomes: A comprehensive meta-analysis 1994-2019

The aim of this study is to comprehensively evaluate quantitative experimental mobile-assisted language learning (MALL) studies published between 1994 and 2019 that meet minimal conditions of research design and statistical analysis. Starting with a bibliographical database of 1,144 references to experimental MALL implementations, of which there were 700 objectively substantiated by quantitative experimental language learning outcomes, only 84 experimental MALL studies met the inclusion requirements. Their analysis addresses two critical sets of research questions. First, what are the general characteristics of the selected studies and, second, what are their language learning outcomes in terms of measured effect size. Nine general characteristics are considered: publication source, chronological distribution, country of origin, institutional environment, sample size, intervention duration, targeted language, language learner competence level, and learning focus. Effect size was calculated separately for between-group (independent, experimental) and within-group (quasi-experimental) treatment studies. In both cases, the overall results were quite large: 0.72 for the former and 1.16 for the latter. An analysis of four critical moderator variables (language learner competence level, language area focus, institutional environment, and intervention duration) revealed similarly large effect sizes. Notwithstanding, analysis of the data also confirmed obvious publication bias and a very high level of heterogeneity that frequently approached 100%. The relevance of positive language learning outcome conclusions thus needs to be tempered by these shortcomings.