A Comparative Study on Pre-Trained Models Based on BERT

The introduction of pre-trained models (PTMs) especially Bidirectional Encoder Representation from Transformer (BERT) [1] brought significant improvements in Natural Language Processing (NLP) tasks and demonstrated the power of transfer learning in large language models. The state-of-the-art performance of BERT on eleven NLP tasks inspired many researchers to focus on building variants based on BERT. This survey is going to collect and investigate the NLP-PTMs researches especially the ones motivated by BERT, concentrating on three main tasks: classifications of their research objects and research methods, and an experimental analysis. The collected papers are going to be classified based on different criteria for each task and provide detailed explanations of why certain research is classified into certain type. In the end, based on the investigation, a future direction for the development of PTMs in NLP is suggested.