Rewire-then-Probe: A Contrastive Recipe for Probing Biomedical Knowledge of Pre-trained Language Models

Knowledge probing is crucial for understanding the knowledge transfer mechanism behind the pre-trained language models (PLMs). Despite the growing progress of probing knowledge for PLMs in the general domain, specialised areas such as biomedical domain are vastly under-explored. To facilitate this, we release a well-curated biomedical knowledge probing benchmark, MedLAMA, constructed based on the Unified Medical Language System (UMLS) Metathesaurus. We test a wide spectrum of state-of-the-art PLMs and probing approaches on our benchmark, reaching at most 3% of acc@10. While highlighting various sources of domain-specific challenges that amount to this underwhelming performance, we illustrate that the underlying PLMs have a higher potential for probing tasks. To achieve this, we propose CONTRASTIVE-P robe, a novel self-supervised contrastive probing approach, that adjusts the underlying PLMs without using any probing data. While CONTRASTIVE-P robe pushes the acc@10 to 24%, the performance gap remains notable. Our human expert evaluation suggests that the probing performance of our CONTRASTIVE-P robe is underestimated as UMLS does not comprehensively cover all existing factual knowledge. We hope MedLAMA and CONTRASTIVE-P robe facilitate further developments of more suited probing techniques for this domain.(1)