BioWiC: An Evaluation Benchmark for Biomedical Concept Representation
Rouhizadeh, H.; Nikishina, I.; Yazdani, A.; Bornet, A.; Zhang, B.; Ehrsam, J.; Gaudet-Blavignac, C.; Naderi, N.; Teodoro, D.
Show abstract
Due to the complexity of the biomedical domain, the ability to capture semantically meaningful representations of terms in context is a long-standing challenge. Despite important progress in the past years, no evaluation benchmark has been developed to evaluate how well language models represent biomedical concepts according to their corresponding context. Inspired by the Word-in-Context (WiC) benchmark, in which word sense disambiguation is reformulated as a binary classification task, we propose a novel dataset, BioWiC, to evaluate the ability of language models to encode biomedical terms in context. We evaluate BioWiC both intrinsically and extrinsically and show that it could be used as a reliable benchmark for evaluating context-dependent embeddings in biomedical corpora. In addition, we conduct several experiments using a variety of discriminative and generative large language models to establish robust baselines that can serve as a foundation for future research.
Matching journals
The top 7 journals account for 50% of the predicted probability mass.
Similar papers in this journal
- A Sequence Labeling Framework for Extracting Drug-Protein Relations from Biomedical Literature 96%
- LSD600: the first corpus of biomedical abstracts annotated with lifestyle–disease relations 95%
- RegulaTome: a corpus of typed, directed, and signed relations between biomedical entities in the scientific literature 94%
Similar papers in this journal
Similar papers in this journal
"Similar papers" are the closest papers from that journal in the model's embedding space. They show what the match is built on, but the ranking comes mostly from a classifier over the whole training set, not from these examples alone.