Concept Drift Monitoring and Diagnostics of Supervised Learning Models via Score Vectors

被引:5
|
作者
Zhang, Kungang [1 ]
Bui, Anh T. [2 ]
Apley, Daniel W. [1 ]
机构
[1] Northwestern Univ, Dept Ind Engn & Management Sci, Evanston, IL 60201 USA
[2] Virginia Commonwealth Univ, Dept Stat Sci & Operat Res, Richmond, VA USA
基金
美国国家科学基金会;
关键词
Control chart; Multivariate EWMA; Predictive model; Score function; CONTROL CHART; TESTS;
D O I
10.1080/00401706.2022.2124310
中图分类号
O21 [概率论与数理统计]; C8 [统计学];
学科分类号
020208 ; 070103 ; 0714 ;
摘要
Supervised learning models are one of the most fundamental classes of models. Viewing supervised learning from a probabilistic perspective, the set of training data to which the model is fitted is usually assumed to follow a stationary distribution. However, this stationarity assumption is often violated in a phenomenon called concept drift, which refers to changes over time in the predictive relationship between covariates X and a response variable Y and can render trained models suboptimal or obsolete. We develop a comprehensive and computationally efficient framework for detecting, monitoring, and diagnosing concept drift. Specifically, we monitor the Fisher score vector, defined as the gradient of the log-likelihood for the fitted model, using a form of multivariate exponentially weighted moving average, which monitors for general changes in the mean of a random vector. In spite of the substantial performance advantages that we demonstrate over popular error-based methods, a score-based approach has not been previously considered for concept drift monitoring. Advantages of the proposed score-based framework include applicability to broad classes of parametric models, more powerful detection of changes as shown in theory and experiments, and inherent diagnostic capabilities for helping to identify the nature of the changes.
引用
收藏
页码:137 / 149
页数:13
相关论文
共 50 条
  • [1] Supervised learning in the presence of concept drift: a modelling framework
    Straat, M.
    Abadi, F.
    Kan, Z.
    Goepfert, C.
    Hammer, B.
    Biehl, M.
    NEURAL COMPUTING & APPLICATIONS, 2022, 34 (01): : 101 - 118
  • [2] Supervised learning in the presence of concept drift: a modelling framework
    M. Straat
    F. Abadi
    Z. Kan
    C. Göpfert
    B. Hammer
    M. Biehl
    Neural Computing and Applications, 2022, 34 : 101 - 118
  • [3] Predictive learning models for concept drift
    Case, J
    Jain, S
    Kaufmann, S
    Sharma, A
    Stephan, F
    ALGORITHMIC LEARNING THEORY, 1998, 1501 : 276 - 290
  • [4] Predictive learning models for concept drift
    Case, J
    Jain, S
    Kaufmann, S
    Sharma, A
    Stephan, F
    THEORETICAL COMPUTER SCIENCE, 2001, 268 (02) : 323 - 349
  • [5] Concept drift detection via competence models
    Lu, Ning
    Zhang, Guangquan
    Lu, Jie
    ARTIFICIAL INTELLIGENCE, 2014, 209 : 11 - 28
  • [6] Explaining Concept Drift of Deep Learning Models
    Wang, Xiaolu
    Wang, Zhi
    Shao, Wei
    Jia, Chunfu
    Li, Xiang
    CYBERSPACE SAFETY AND SECURITY, PT II, 2019, 11983 : 524 - 534
  • [7] Maintaining and Monitoring AIOps Models Against Concept Drift
    Poenaru-Olaru, Lorena
    Cruz, Luis
    Rellermeyer, Jan S.
    van Deursen, Arie
    2023 IEEE/ACM 2ND INTERNATIONAL CONFERENCE ON AI ENGINEERING - SOFTWARE ENGINEERING FOR AI, CAIN, 2023, : 98 - 99
  • [8] Monitoring Concept Drift in Continuous Federated Learning Platforms
    Dusing, Christoph
    Cimiano, Philipp
    ADVANCES IN INTELLIGENT DATA ANALYSIS XXII, PT II, IDA 2024, 2024, 14642 : 83 - 94
  • [9] Particle Competition and Cooperation in Networks for Semi-Supervised Learning with Concept Drift
    Breve, Fabricio
    Zhao, Liang
    2012 INTERNATIONAL JOINT CONFERENCE ON NEURAL NETWORKS (IJCNN), 2012,
  • [10] Semi-supervised Ensemble Learning of Data Streams in the Presence of Concept Drift
    Ahmadi, Zahra
    Beigy, Hamid
    HYBRID ARTIFICIAL INTELLIGENT SYSTEMS, PT II, 2012, 7209 : 526 - 537