PAG-LLM: Paraphrase and Aggregate with Large Language Models for Minimizing Intent Classification Errors

Large language models (LLM) have achieved remarkable success in natural language generation but lesser focus has been given to their applicability in key tasks such as intent-classification. We show that LLMs like LLaMa can achieve high performance on intent classification tasks with large number of classes but still make classification errors and worse, generate out-of-vocabulary intent labels. To address these critical issues, we introduce Paraphrase and AGgregate (PAG)-LLM approach wherein an LLM generates multiple paraphrases of the input query (parallel queries), performs intent classification for the original query and each paraphrase, and at the end aggregate all the predicted intent labels based on their confidence scores. We evaluate PAG-LLM on two large intent classification datasets: CLINC, and Banking and show 22.7% and 15.1% error reduction. We showthat PAG-LLM is especially effective for hard examples where LLM is uncertain, and reduces the critical misclassification and hallucinated label generation errors.