The stability of prediction precision under complex loading paths is one of the key challenges in the task of multiaxial fatigue life prediction. This study addresses the challenges of unstable prediction precision in machine learning models, while further improving the precision of multiaxial fatigue life prediction. A novel neural network based on a transformer framework is proposed to capture dependencies between data at multiple scales. Meanwhile, physical loss function with soft adjustments is proposed to add physical constraints to the proposed neural network. These two mechanisms assist each other in improving the accuracy and stability of fatigue life prediction. Performance validation was conducted using fatigue data from nine distinct materials. Comparative analysis was performed against six existing models to evaluate the efficacy of the proposed physical neural network. Experimental evidence supports the high predictive accuracy of the proposed physical neural network, which also demonstrates robust stability across diverse conditions.