Recognizing and Predicting Business Communication Outcomes Using Local LLMs

In this paper, we use machine learning methods based on three popular open-source large language models (LLMs) that can run efficiently on local computers to recognize and predict business communication outcomes. The methods include zero-shot Alpaca-Lora-7B, BigBird with fine-tuning, Alpaca-Lora-7B with prompting and LoRA-based fine-tuning, and Llama2-70B-Chat with one-stage and two-stage prompting. Our experimental results on a real-world dataset showed promising results of LLMs for both communication outcome recognition and prediction tasks. On recognizing communication outcomes, Alpaca-Lora-7B with prompt engineering and LoRa-based fine-tuning performed the best achieving 94.66% accuracy while BigBird with fine-tuning is closely behind with 94.27% accuracy, which, in turn, outperformed prompt engineering on LLMs. On predicting communication outcomes based on partial conversations, BigBird fine-tuned using the beginning 70% of each conversations achieved more than 85% prediction accuracy based on the first 70% of each conversation.