Using Noise and External Knowledge to Enhance Chinese Pre-trained Model

Pre-trained language models (PLMs) have the risk of overfitting pre-trained tasks and data in fine-tuning, while Chinese PLMs often ignore external knowledge such as word and sentence to learn representations. Therefore, we propose a Chinese PLM enhancement method using noise and external knowledge (NEK). NEK first adds different uniform noises to the PLM according to the standard deviation of different parameter matrices, so as to obtain the perturbed PLM. In the fine-tuning phase, NEK builds a heterogeneous linguistic graph based on external knowledge. This module adopts a graph-based approach to generalize information of different granularities in Chinese linguistics, and enhances Chinese PLM on this basis. Experimental results show that NEK brings performance improvements to a variety of different Chinese PLMs on six natural language processing tasks on eight benchmark datasets.