Multi-document summarization through subsection-aware pre-training objectives

Pre-training language models are increasingly being utilized for multi-document summarization (MDS) tasks. However, pre-training typically necessitates large-scale, domain-specific data. Most MDS pre-training models process multiple documents as a single document, while ignoring the subsection relationship information among different documents. In this paper, we focus on pre-training objectives for MDS, which assume key information appears across multiple documents that point to the same topic. We segment each document into several subsections based on text structure features. Then, we compare the subsections of different documents, extract key sentences from these subsections using text similarity, and generate a proxy summary. Experimental results on Multi-News and WikiSum demonstrate that our proposed model outperforms compared MDS models in terms of ROUGE scores and maintains strong performance even with limited data samples.