Stable On-line Learning with Optimized Local Learning, but Minimal Change of the Global Output

This work presents a novel approach to on-line learning regression. The well-known risk functional is formulated in an incremental manner that is aggressive to incorporate a new example locally as much as possible and at the same time passive in the sense that the overall output is changed as little as possible. To achieve this localized learning, knowledge about the model structure of the approximator is utilized to steer the adaptation of the parameter vector. We present a continuously adapting first order learning algorithm that is stable, even for complex model structures and low data densities. Additionally, we present an approach to extend this algorithm to a second order version with greater robustness but lower flexibility. Both algorithms are compared to state of the art methods as well on synthetic data as on benchmark datasets to show the benefits of the new approach.