Infusing Dependency Syntax Information into a Transformer Model for Document-Level Relation Extraction from Biomedical Literature

In biomedical domain, document-level relation extraction is a challenging task that offers a new and more effective approach for long and complex text mining. Studies have shown that the Transformer models the dependencies of any two tokens without regard to their syntax-level dependency in the sequence. In this work, we propose a Dependency Syntax Transformer Model, i.e., the DSTM model, to improve the Transformer's ability in long-range modeling dependencies. Three methods are proposed for introducing dependency syntax information into the Transformer to enhance the attention of tokens with dependencies in a sentence. The dependency syntax Transformer model improves the Transformer's ability to handle long text in document-level relation extraction. Our experimental results on the document-level relation extraction dataset CDR in the biomedical field prove the validity of the DSTM model, and the experimental results on the generic domain dataset DocRED prove the universality.