Fine-Tuning Transformer-Based Representations in Active Learning for Labelling Crisis Dataset of Tweets

Supervised machine learning-based models are generally used for classifying tweets related to crisis. A labelled tweet dataset is a major requirement for training the models. Labelling huge quantities of text data manually is a time-consuming and costly process. Active learning reduces some of the work necessary to use vast volumes of unlabelled data for machine learning tasks without fully labelling them. During the active learning process, the representation strategy employed for tweets has a substantial impact on the process’ effectiveness. The representations like Bag-of-Words and representations based on pre-trained word embeddings like GloVe have been used in the active learning process and have proven to be effective in representing tweets. The introduction of pre-trained transformer-based models like BERT, XLNet, and GPT-2 is prevalent in natural language processing tasks. These transformer-based models can also be used to represent embeddings of tweets but are not yet explored fully as an alternative to other embeddings used in active learning. This work offers a complete evaluation of the usefulness of representations for active learning, based on transformer-based language models. This study also demonstrates that transformer-based models, particularly BERT-like models, which have yet to be widely used in active learning, outperform more regularly used vector representations such as Bag-of-Words or other traditional word-embeddings such as GloVe. This work also compares the usefulness of representations based on the “[CLS]” token and aggregated representations generated using BERT-like models. The effectiveness of representations based on different types of BERT such as DistilBert, Roberta, and Albert is also investigated in this work. In this work finally, we propose a method called adaptive fine-tuning active learning, which is fine-tuning the representations produced by BERT-like models during the active learning process. The results show that the minimal label information gained through active learning may be utilized to not only train a classifier but also adaptively improve embedding produced by BERT-like transformer-based language models. © 2023, The Author(s), under exclusive licence to Springer Nature Singapore Pte Ltd.