Physical activation functions (PAFs): An approach for more efficient induction of physics into physics-informed neural networks (PINNs)

In recent years, the evolution of Physics-Informed Neural Networks (PINNs) has reduced the gap between Deep Learning (DL) based methods and analytical/numerical approaches in scientific computing. However, there are still complications in training PINNs and the optimal interleaving of physical models. In this work, we introduce the concept of Physical Activation Functions (PAFs), in which one can use generic AFs with their mathematical expression inherited from the physical description of the evaluated system, instead of solely usage of standard activation functions (AFs) such as tanh, and sigmoid for all the neurons. The expression of PAFs could be selected based on individual terms appearing in the analytical solution, the initial or boundary conditions of the PDE system, or a component in the composition-of-functions type solutions. The PAFs could be applied in NNs, either in explicit, or self-adaptive forms. In the explicit approach, the main activation function of the network is replaced by PAF in some of the neurons of the network. In the self-adaptive implementation approach, the relative impact of PAFs (compared to the base AF) for each neuron was determined automatically. We tested the performance of PAFs in both forward and inverse problems for several PDEs, such as 1D and 2D wave equations, the Advection-Convection equation, the 1D heterogeneous, and 2D diffusion equations, and the Laplace equation. The main advantage of PAFs, compared to using standard AFs, was the more efficient constraining and interleaving of PINNs with the physical phenomena and their underlying mathematical models. The added PAFs significantly improved the predictions of PINNs for the testing data that were out-of-training distribution. Furthermore, applying PAFs reduced the size of the PINNs by up to 75 % in different cases while maintaining the same accuracy. Also, the training process was improved by reducing the value of the total loss term by one to two orders of magnitude. Furthermore, it improved the precision of the calculated properties in the examined inverse problems, for both clean and noisy observational data. It can be concluded that using PAFs helps in generating PINNs with less complexity and more validity for longer ranges of prediction.