Machine and deep learning performance in out-of-distribution regressions

被引:0
作者
Shmuel, Assaf [1 ]
Glickman, Oren [1 ]
Lazebnik, Teddy [2 ]
机构
[1] Bar Ilan Univ, Dept Comp Sci, Ramat Gan, Israel
[2] UCL, Canc Inst, Dept Canc Biol, London, England
来源
MACHINE LEARNING-SCIENCE AND TECHNOLOGY | 2024年 / 5卷 / 04期
关键词
data-driven model generalization; out of distribution; feature engineering; symbolic regression; machine learning robustness; SYMBOLIC REGRESSION; BIG DATA; MODEL;
D O I
10.1088/2632-2153/ada221
中图分类号
TP18 [人工智能理论];
学科分类号
081104 ; 0812 ; 0835 ; 1405 ;
摘要
Machine learning (ML) and deep learning (DL) models are gaining popularity due to their effectiveness in many computational tasks. These models are based on an intuitive, but frequently unsatisfied, assumption that the data used to train these models is well-representing the task at hand. This gives rise to the out-of-distribution (OOD) challenge which can cause an unexpected drop in the data-driven model's performance. In this study, we evaluate the performance of various ML and DL models in in-distribution (ID) versus OOD prediction. While the degradation in OOD performance is well acknowledged, to the best of our knowledge, this is one of the first studies to quantify it for various models on a large benchmark n = 15 real-world regression datasets. We extensively ( n>40000 runs) compare the ID versus OOD performance of XGBoost, random forest, K-nearest-neighbors, support vector machine, and linear regression models, as well as AutoML models (Tree-based Pipeline Optimization Tool and AutoKeras). In addition, to tackle this challenge, we propose to integrate a symbolic regression (SR) as a feature engineering method model with an ML or DL model to improve its performance for OOD samples. Our results show that the incorporation of SR-derived features significantly enhances the predictive capabilities of both ML and DL models with 3.70% and 10.20%, on average, of the OOD samples, respectively, without reducing ID performance and in fact improving it to a slightly lower extent. As such, this method can help produce more generalized and robust data-driven models.
引用
收藏
页数:30
相关论文
共 50 条
[31]   ROTOGBML: TOWARDS OUT-OF-DISTRIBUTION GENERALIZATION FOR GRADIENT-BASED META-LEARNING [J].
Zhang, Min ;
Zhuang, Zifeng ;
Wang, Zhitao ;
Wang, Donglin .
2024 IEEE INTERNATIONAL CONFERENCE ON MULTIMEDIA AND EXPO, ICME 2024, 2024,
[32]   Learning to Improve Out-of-Distribution Generalization via Self-Adaptive Language Masking [J].
Jiang, Shuoran ;
Pan, Youcheng ;
Chen, Qingcai ;
Xiang, Yang ;
Wu, Xiangping .
IEEE-ACM TRANSACTIONS ON AUDIO SPEECH AND LANGUAGE PROCESSING, 2024, 32 :2739-2750
[33]   OUT-OF-DISTRIBUTION DETECTION FOR LEARNING-BASED CHEST X-RAY DIAGNOSIS [J].
Chen, Wenlong ;
Feng, Chuanwen ;
Ke, Ao ;
Xie, Xike ;
Zhou, S. Kevin .
2024 IEEE INTERNATIONAL CONFERENCE ON ACOUSTICS, SPEECH AND SIGNAL PROCESSING, ICASSP 2024, 2024, :1816-1820
[34]   SEMI-SUPERVISED LEARNING WITH OUT-OF-DISTRIBUTION UNLABELED SAMPLES FOR RETINAL IMAGE CLASSIFICATION [J].
Jia, Lize ;
Guo, Jia ;
Zhang, Weihang ;
Liu, Hanruo ;
Wang, Ningli ;
Li, Huiqi .
2023 IEEE 20TH INTERNATIONAL SYMPOSIUM ON BIOMEDICAL IMAGING, ISBI, 2023,
[35]   Improving out-of-distribution detection by enforcing confidence margin [J].
Tamang, Lakpa ;
Bouadjenek, Mohamed Reda ;
Dazeley, Richard ;
Aryal, Sunil .
KNOWLEDGE AND INFORMATION SYSTEMS, 2025, 67 (07) :5541-5569
[36]   An Out-of-Distribution Attack Resistance Approach to Emotion Categorization [J].
Shehu H.A. ;
Browne W.N. ;
Eisenbarth H. .
IEEE Transactions on Artificial Intelligence, 2021, 2 (06) :564-573
[37]   PixOOD: Pixel-Level Out-of-Distribution Detection [J].
Vojir, Tomas ;
Sochman, Jan ;
Matas, Jiri .
COMPUTER VISION - ECCV 2024, PT LX, 2025, 15118 :93-109
[38]   Out-of-Distribution Detection with Logical Reasoning (Extended Abstract) [J].
Kirchheim, Konstantin ;
Gonschorek, Tim ;
Ortmeier, Frank .
KI 2024: ADVANCES IN ARTIFICIAL INTELLIGENCE, KI 2024, 2024, 14992 :346-349
[39]   Adversarial Training for Ultrasound Beamforming in Out-of-Distribution Scenarios [J].
Salazar-Reque, Itamar ;
Juarez, Jesus ;
Lavarello, Roberto .
2024 IEEE ULTRASONICS, FERROELECTRICS, AND FREQUENCY CONTROL JOINT SYMPOSIUM, UFFC-JS 2024, 2024,
[40]   Wafer defect pattern classification with detecting out-of-distribution [J].
Kim, Yusung ;
Cho, Donghee ;
Lee, Jee-Hyong .
MICROELECTRONICS RELIABILITY, 2021, 122