Intermediate Task Ensembling for Sarcasm Detection

Navigating nuanced linguistic features through machine learning plays an important role in academic, private, and public sectors as researchers and developers use computational methods to parse, process, and understand an ever-increasing volume of natural language text. While recent advances in natural language processing and large language models have improved access to state-of-the-art performance, partly through the evolution of pre-training and fine-tuning processes, some particularly difficult linguistic tasks, such as sarcasm detection, remain to be solved. Such tasks can be highly disruptive to computational systems that rely on natural language processing. In this paper, we approach sarcasm detection by leveraging the RoBERTa model, a two-step fine-tuning process called Intermediate Fine-Tuning, and ensembling theory. We establish baselines, create ensembles, explore ensemble predictions, and analyze both baseline and ensemble performance. This research shows that intermediate fine-tuning can create sufficiently performant and diverse inducers for ensembles, and that those ensembles may also outperform single-model baselines on sarcasm detection tasks.