Sentence-graph-level knowledge injection with multi-task learning

Language representation learning is a fundamental task for natural language understanding. It aims to represent natural language sentences and classify their mentioned entities and relations, which usually requires injecting external entity and relation knowledge into sentence representation. Existing methods typically inject factual knowledge into pre-trained language models by sequentially concatenating knowledge behind the sentence, with less attention to the structured information from the knowledge graph and the interactive relationship within. In this paper, we learn the sentence representation based on both Sentence- and Graph- level knowledge at the fine-tuning stage with a multi-task learning framework (SenGraph). At sentence-level, we concatenate factual knowledge with the sentence by a sequential structure, and train it with a sentence-level task. At the graph-level, we construct all the knowledge and sentence information as a graph, and introduce a relational GAT to inject useful knowledge into sentences selectively. Meanwhile, we design two graph-based auxiliary tasks to align the heterogeneous embedding space between the natural language sentence and the knowledge graph. We evaluate our model on four knowledge-driven benchmark datasets. The experimental results demonstrate the effectiveness of the proposed method using less computational resources.