Learning under (1+ε)-moment conditions

We study the theoretical underpinning of a robust empirical risk minimization (RERM) scheme which has been finding numerous successful applications across various data science fields owing to its robustness to outliers and heavy-tailed noises. The specialties of RERM lie in its nonconvexity and that it is induced by a loss function with an integrated scale parameter trading off the robustness and the prediction accuracy. The nonconvexity of RERM and the integrated scale parameter also bring barriers when assessing its learning performance theoretically. In this paper, concerning the study of RERM, we make the following main contributions. First, we establish a no-free-lunch result, showing that there is no hope of distribution-free learning of the truth without adjusting the scale parameter. Second, by imposing the (1 + epsilon)-th (with epsilon > 0) order moment condition on the response variable, we establish a comparison theorem that characterizes the relation between the excess generalization error of RERM and its prediction error. Third, with a diverging scale parameter, we establish almost sure convergence rates for RERM under the (1 + epsilon)-moment condition. Notably, the (1 + epsilon)-moment condition allows the presence of noise with infinite variance. Last but not least, the learning theory analysis of RERM conducted in this study, on one hand, showcases the merits of RERM on robustness and the trade-off role that the scale parameter plays, and on the other hand, brings us inspirational insights into robust machine learning. Published by Elsevier Inc.