On the effectiveness of small, discriminatively pre-trained language representation models for biomedical text mining
Ozyurt, I. B.
Show abstract
Neural language representation models such as BERT [1] have recently shown state of the art performance in downstream NLP tasks and bio-medical domain adaptation of BERT (Bio-BERT [2]) has shown same behavior on biomedical text mining tasks. However, due to their large model size and resulting increased computational need, practical application of models such as BERT is challenging making smaller models with comparable performance desirable for real word applications. Recently, a new language transformers based language representation model named ELECTRA [3] is introduced, that makes efficient usage of training data in a generative-discriminative neural model setting that shows performance gains over BERT. These gains are especially impressive for smaller models. Here, we introduce a small ELECTRA based model named Bio-ELECTRA that is eight times smaller than BERT BASE and achieves comparable performance on biomedical question answering and yes/no question answer classification tasks. The model is pre-trained from scratch on PubMed abstracts using a consumer grade GPU with only 8GB memory. For biomedical named entity recognition, however, large BERT Base model outperforms both Bio-ELECTRA and ELECTRA-Small++.
Matching journals
The top 8 journals account for 50% of the predicted probability mass.
Similar papers in this journal
Similar papers in this journal
- BERTMeSH: Deep Contextual Representation Learning for Large-scale High-performance MeSH Indexing with Full Text 95%
- Variomes: a high recall search engine to support the curation of genomic variants 94%
- CoCoScore: Context-aware co-occurrence scoring for text mining applications using distant supervision 94%
Similar papers in this journal
Similar papers in this journal
Similar papers in this journal
- Optimizing biomedical information retrieval with a keyword frequency-driven Prompt Enhancement Strategy 96%
- Relation extraction between bacteria and biotopes from biomedical texts with attention mechanisms and domain-specific contextual representations 96%
- pyMeSHSim: an integrative python package to realize biomedical named entity recognition, normalization and comparison 93%
"Similar papers" are the closest papers from that journal in the model's embedding space. They show what the match is built on, but the ranking comes mostly from a classifier over the whole training set, not from these examples alone.