Enhancing text generation from knowledge graphs with cross-structure attention distillation

Existing Large-scale pre-trained language models (PLMs) can effectively enhance the knowledge-graph-to-text (KG-to-text) generation by processing the linearized version of a graph. However, recent work ignores the effective interaction between the linear representation of the knowledge graphs and the generated texts. To address this problem, we propose a distillation model that utilizes a cross-structure attention mechanism for generating texts from knowledge graphs. This mechanism is designed to obtain rich contextual semantic representations between linearized structured data and text information in the teacher model. And then, we train a student model to mimic the behavior of the teacher model by outputting logits with online distillation. Experimental results demonstrate that our distillation model outperforms many pre-trained natural language generation (NLG) models on various KG-to-text datasets.