Multi-Objective Loss Balancing for Physics-Informed Deep Learning

Physics-Informed Neural Networks (PINN) are deep learning algorithms that leverage physical laws by including partial differential equations together with a respective set of boundary and initial conditions as penalty terms in their loss function. In this work, we observe the significant role of correctly weighting the combination of multiple competitive loss functions for training PINNs effectively. To this end, we implement and evaluate different methods aiming at balancing the contributions of multiple terms of the PINN's loss function and their gradients. After reviewing three existing loss scaling approaches (Learning Rate Annealing, GradNorm and SoftAdapt), we propose a novel self-adaptive loss balancing scheme for PINNs named ReLoBRaLo (Relative Loss Balancing with Random Lookback). We extensively evaluate the performance of the aforementioned balancing schemes by solving both forward as well as inverse problems on three benchmark PDEs for PINNs: Burgers' equation, Kirchhoff's plate bending equation, Helmholtz's equation and over 20 PDEs from the "PINNacle" collection. The results show that ReLoBRaLo is able to consistently outperform the baseline of existing scaling methods in terms of accuracy while also inducing significantly less computational overhead for a variety of PDE classes.