CoLRP: A Contrastive Learning Abstractive Text Summarization Method with ROUGE Penalty

Contrastive learning can reduce the impact of exposure bias associated with training using maximum likelihood estimation, which aims to pull together positive samples to increase the likelihood of high-quality summaries and push away irrelevant negative samples to reduce the likelihood of low-quality summaries. In contrastive learning-based text summarization methods, a standard method for selecting positive and negative samples is randomly selected within a batch. This method can lead to sampling bias to the extent that the consistency of the representation space is compromised. Therefore, we propose a new method to penalize false negatives based on ROUGE metric scores as weights to sample from the dynamic output of the model training process. The method calculates ROUGE metric scores for penalizing false negatives in real-time and can distinguish between positive and negative samples to ensure spatial consistency and alleviate exposure bias. Experimental results on XSum, CNN/DM, and Multi-News datasets show that our approach effectively improves the performance of the latest text summarization pre-training models.