A Multi-teacher Knowledge Distillation Framework for Distantly Supervised Relation Extraction with Flexible Temperature

Distantly supervised relation extraction (DSRE) generates large-scale annotated data by aligning unstructured text with knowledge bases. However, automatic construction methods cause a substantial number of incorrect annotations, thereby introducing noise into the training process. Most sentence-level relation extraction methods rely on filters to remove noise instances, meanwhile, they ignore some useful information in negative instances. To effectively reduce noise interference, we propose a Multi-teacher Knowledge Distillation framework for Relation Extraction (MKDRE) to extract semantic relations from noisy data based on both global information and local information. MKDRE addresses two main problems: the deviation in knowledge propagation of a single teacher and the limitation of traditional distillation temperature on information utilization. Specifically, we utilize flexible temperature regulation (FTR) to adjust the temperature assigned to each training instance, so as to dynamically capture local relations between instances. Furthermore, we introduce information entropy of hidden layers to gain stable temperature calculations. Finally, we propose multi-view knowledge distillation (MVKD) to express global relations among teachers from various perspectives to gain more reliable knowledge. The experimental results on NYT19-1.0 and NYT19-2.0 datasets show that our proposed MKDRE significantly outperforms previous methods in sentence-level relation extraction.