A Meta-Learning Approach to Predicting Performance and Data Requirements

被引:5
|
作者
Jain, Achin [1 ]
Swaminathanl, Gurumurthy [1 ]
Favarol, Paolo [1 ]
Yang, Hao [1 ]
Ravichandrae, Avinash [1 ]
Harutyunyan, Hrayr [1 ,2 ]
Achillel, Alessandro [1 ]
Dabeerl, Onkar [1 ]
Schiele, Bernt [1 ]
Swaminathan, Ashwin [1 ]
Soata, Stefano [1 ]
机构
[1] AWS AI Labs, Seattle, WA 77002 USA
[2] Univ Southern Calif, Los Angeles, CA USA
来源
2023 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION, CVPR | 2023年
关键词
BENCHMARK;
D O I
10.1109/CVPR52729.2023.00353
中图分类号
TP18 [人工智能理论];
学科分类号
081104 ; 0812 ; 0835 ; 1405 ;
摘要
We propose an approach to estimate the number of samples required for a model to reach a target performance. We find that the power law, the de facto principle to estimate model performance, leads to a large error when using a small dataset (e.g., 5 samples per class) for extrapolation. This is because the log-performance error against the log-dataset size follows a nonlinear progression in the few-shot regime followed by a linear progression in the high-shot regime. We introduce a novel piecewise power law (PPL) that handles the two data regimes differently. To estimate the parameters of the PPL, we introduce a random forest regressor trained via meta learning that generalizes across classification/detection tasks, ResNet/ViT based architectures, and random/pre-trained initializations. The PPL improves the performance estimation on average by 37% across 16 classification and 33% across 10 detection datasets, compared to the power law. We further extend the PPL to provide a confidence bound and use it to limit the prediction horizon that reduces over-estimation of data by 76% on classification and 91% on detection datasets.
引用
收藏
页码:3623 / 3632
页数:10
相关论文
共 50 条
  • [31] A Meta-Learning Approach for Custom Model Training
    Eshratifar, Amir Erfan
    Abrishami, Mohammad Saeed
    Eigen, David
    Pedram, Massoud
    THIRTY-THIRD AAAI CONFERENCE ON ARTIFICIAL INTELLIGENCE / THIRTY-FIRST INNOVATIVE APPLICATIONS OF ARTIFICIAL INTELLIGENCE CONFERENCE / NINTH AAAI SYMPOSIUM ON EDUCATIONAL ADVANCES IN ARTIFICIAL INTELLIGENCE, 2019, : 9937 - 9938
  • [32] On performance of Meta-learning Templates on Different Datasets
    Kordik, Pavel
    Cerny, Jan
    2012 INTERNATIONAL JOINT CONFERENCE ON NEURAL NETWORKS (IJCNN), 2012,
  • [33] A Meta-Learning Architecture based on Linked Data
    Dos Santos, Ricardo
    Aguilar, Jose
    Puerto, Eduard
    2021 XLVII LATIN AMERICAN COMPUTING CONFERENCE (CLEI 2021), 2021,
  • [34] On the Accuracy of Meta-learning for Scalable Data Mining
    Chan P.K.
    Stolfo S.J.
    Journal of Intelligent Information Systems, 1997, 8 (1) : 5 - 28
  • [35] PREDICTING ABNORMAL STOCK RETURN VOLATILITY USING TEXTUAL ANALYSIS OF NEWS - A META-LEARNING APPROACH
    Myskova, Renata
    Hajek, Petr
    Olej, Vladimir
    AMFITEATRU ECONOMIC, 2018, 20 (47) : 185 - 201
  • [36] Meta-data: Characterization of input features for meta-learning
    Castiello, C
    Castellano, G
    Fanelli, AM
    MODELING DECISIONS FOR ARTIFICIAL INTELLIGENCE, PROCEEDINGS, 2005, 3558 : 457 - 468
  • [37] Active meta-learning for predicting and selecting perovskite crystallization experiments
    Shekar, Venkateswaran
    Nicholas, Gareth
    Najeeb, Mansoor Ani
    Zeile, Margaret
    Yu, Vincent
    Wang, Xiaorong
    Slack, Dylan
    Li, Zhi
    Nega, Philip W.
    Chan, Emory M.
    Norquist, Alexander J.
    Schrier, Joshua
    Friedler, Sorelle A.
    JOURNAL OF CHEMICAL PHYSICS, 2022, 156 (06):
  • [38] Predicting Scores of Medical Imaging Segmentation Methods with Meta-learning
    van Sonsbeek, Tom
    Cheplygina, Veronika
    INTERPRETABLE AND ANNOTATION-EFFICIENT LEARNING FOR MEDICAL IMAGE COMPUTING, IMIMIC 2020, MIL3ID 2020, LABELS 2020, 2020, 12446 : 242 - 253
  • [39] Learning unbiased classifiers from biased data with meta-learning
    Ragonesi, Ruggero
    Morerio, Pietro
    Murino, Vittorio
    2023 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION WORKSHOPS, CVPRW, 2023, : 1 - 9
  • [40] Predicting heave and pitch motions of an FPSO using meta-learning
    Liu, Yue
    Zhang, Xiantao
    Dong, Qing
    Guo, Xiaoxian
    Tian, Xinliang
    Chen, Gang
    MARINE STRUCTURES, 2024, 98