Drug-drug interaction extraction from biomedical texts based on multi-attention mechanism

With the recent revival of interest in deep neural networks, many studies have focused on exploring deep learning methods for automatic DDIs (Drug-drug Interactions) extraction. However, due to the "black box" nature of deep neural networks, the researches on discussing the interpretability and reliability of the output from the models still fall short. Considering the interpretability of the output, we propose a model based on mechanism for DDIs extraction. Our model is able to compute the importance scores in the neural network, which can be utilized to measure the contributions of different words while the model makes decision. Besides, a key feature of our model is that we design a classified tag word, named as [CLS], to learn the global information for DDI classification. Our model consists of three layers, including Bi-LSTM layer, attention layer and dense Layer. The Bi-LSTM layer uses a bi-directional long short term memory network to learn embeddings for each word, also for the [CLS] tag. We equip the attention layer with a multi-head attention mechanism to learn different contributions of each work for the classified tag word. In the dense layer, instead of performing pooling to capture global information across the whole sentence, we combine two [CLS] tags produced by the Bi-LSTM layer and the attention layer, respectively. At last, we use a Softmax function for classification. The experimental results have shown that our model has competitive advantage in extract the relation between the two candidate drugs in one instance. By visualizing the attention weights with the corresponding instance, it offers an intuitive way to understand the basis of the classification results. Moreover, we use the [CLS] tag to learn the global information instead of performing pooling in the dense layer, which can reduce the training time and also improves performance.