Enhancing Legal Text Entailment with Prompt-Based ChatGPT: An Empirical Study

This research paper focuses on the task of legal text entailment, which involves determining whether a given statement logically follows from the facts stated in a legal text. In this paper, we perform experiments with ChatGPT, a large language model developed by OpenAI, for the task of legal text entailment. Among various prompt settings, we find that by using appropriate prompts while asking ChatGPT to output step-by-step reasoning, ChatGPT outperforms previous approaches by a large margin in the COLIEE 2022 dataset, achieving an improvement of up to 10.09% absolute. We also conduct an extensive analysis of how the model makes incorrect predictions, providing insights for potential improvements in future work. This research demonstrates the potential of using state-of-the-art natural language processing models, such as ChatGPT, to address complex legal tasks and advance the field of automated legal text entailment.