Flexible Classification, Question-Answering and Retrieval with Siamese Neural Networks for Biomedical Texts

Training transformers models on biomedical data has shown promising results. However, these language models require fine-tuning large models on very specific supervised data for each task. In this paper, we propose to use siamese neural models (sentence transformers) that embed texts to be compared in a vector space, and apply them to the biomedical domain on three main tasks: classification, question answering and retrieval. Training is based on articles from the MEDLINE bibliographic database associated with their MeSH (Medical Subject Headings) keywords and optimizes an objective self-supervised contrastive learning function. The representation of the texts (embeddings) obtained by our siamese models can be stored, indexed and used with transfer learning without needing the language models. The obtained results on several benchmarks show that the proposed models can solve these tasks with results comparable to biomedical cross-encoders transformers and offer a several advantages by being flexible and efficient at inference time. (Models and data available: https://github.com/arieme/BioSTransformers.git).