DrBERT: A Robust Pre-trained Model in French for Biomedical and Clinical domains
Labrak, Y.; Bazoge, A.; Dufour, R.; Rouvier, M.; Morin, E.; Daille, B.; Gourraud, P.-A.
Show abstract
In recent years, pre-trained language models (PLMs) achieve the best performance on a wide range of natural language processing (NLP) tasks. While the first models were trained on general domain data, specialized ones have emerged to more effectively treat specific domains. In this paper, we propose an original study of PLMs in the medical domain on French language. We compare, for the first time, the performance of PLMs trained on both public data from the web and private data from healthcare establishments. We also evaluate different learning strategies on a set of biomedical tasks. In particular, we show that we can take advantage of already existing biomedical PLMs in a foreign language by further pre-train it on our targeted data. Finally, we release the first specialized PLMs for the biomedical field in French, called DrBERT, as well as the largest corpus of medical data under free license on which these models are trained.
Matching journals
The top 7 journals account for 50% of the predicted probability mass.
Similar papers in this journal
Similar papers in this journal
- A Sequence Labeling Framework for Extracting Drug-Protein Relations from Biomedical Literature 93%
- LSD600: the first corpus of biomedical abstracts annotated with lifestyle–disease relations 92%
- RegulaTome: a corpus of typed, directed, and signed relations between biomedical entities in the scientific literature 92%
Similar papers in this journal
- Comparing neural language models for medical concept representation and patient trajectory prediction 93%
- Enriching Representation Learning Using 53 Million Patient Notes through Human Phenotype Ontology Embedding 93%
- Graph Neural Network Modelling as a potentially effective Method for predicting and analyzing Procedures based on Patient Diagnoses 93%
Similar papers in this journal
"Similar papers" are the closest papers from that journal in the model's embedding space. They show what the match is built on, but the ranking comes mostly from a classifier over the whole training set, not from these examples alone.