Back

A Comprehensive Benchmark Study on Biomedical Text Generation and Mining with ChatGPT

Chen, Q.; Sun, H.; Liu, H.; Jiang, Y.; Ran, T.; Jin, X.; Xiao, X.; Lin, Z.; Niu, Z.; Chen, H.

2023-04-20 bioinformatics
10.1101/2023.04.19.537463 bioRxiv
Show abstract

In recent years, the development of natural language process (NLP) technologies and deep learning hardware has led to significant improvement in large language models(LLMs). The ChatGPT, the state-of-the-art LLM built on GPT-3.5, shows excellent capabilities in general language understanding and reasoning. Researchers also tested the GPTs on a variety of NLP related tasks and benchmarks and got excellent results. To evaluate the performance of ChatGPT on biomedical related tasks, this paper presents a comprehensive benchmark study on the use of ChatGPT for biomedical corpus, including article abstracts, clinical trials description, biomedical questions and so on. Through a series of experiments, we demonstrated the effectiveness and versatility of Chat-GPT in biomedical text understanding, reasoning and generation.

Matching journals

The top 6 journals account for 50% of the predicted probability mass.

1
Bioinformatics
◐ 1204 papers in training set
Top 1%
18.4%
Similar papers in this journal
2
Bioinformatics Advances
◐ 203 papers in training set
Top 0.4%
7.8%
Similar papers in this journal
3
Artificial Intelligence in Medicine
○ 17 papers in training set
Top 0.1%
6.7%
Similar papers in this journal
4
BioData Mining
○ 22 papers in training set
Top 0.1%
6.7%
Similar papers in this journal
5
Computers in Biology and Medicine
○ 128 papers in training set
Top 0.5%
6.2%
6
Database
◐ 61 papers in training set
Top 0.1%
6.2%
Similar papers in this journal
50% of probability mass above
7
Journal of Biomedical Informatics
○ 47 papers in training set
Top 0.3%
5.4%
8
BMC Bioinformatics
○ 457 papers in training set
Top 2%
4.3%
9
GigaScience
◐ 212 papers in training set
Top 1%
2.6%
10
Briefings in Bioinformatics
◐ 354 papers in training set
Top 3%
2.4%
11
PLOS ONE
● 5266 papers in training set
Top 45%
1.9%
12
Frontiers in Genetics
○ 230 papers in training set
Top 3%
1.7%
13
Frontiers in Bioinformatics
○ 49 papers in training set
Top 0.5%
1.7%
14
Computational and Structural Biotechnology Journal
● 242 papers in training set
Top 3%
1.7%
15
IEEE/ACM Transactions on Computational Biology and Bioinformatics
● 38 papers in training set
Top 0.6%
1.7%
16
Scientific Reports
○ 3612 papers in training set
Top 54%
1.7%
17
BMC Medical Informatics and Decision Making
○ 43 papers in training set
Top 1%
1.7%
18
Journal of the American Medical Informatics Association
◐ 71 papers in training set
Top 2%
1.5%
19
Scientific Data
○ 209 papers in training set
Top 2%
1.3%
20
iScience
○ 1154 papers in training set
Top 27%
1.1%
21
JAMIA Open
◐ 42 papers in training set
Top 1%
1.1%
22
IEEE Transactions on Computational Biology and Bioinformatics
● 20 papers in training set
Top 0.5%
0.9%
23
PLOS Digital Health
● 106 papers in training set
Top 4%
0.8%
24
Journal of Computational Biology
○ 48 papers in training set
Top 1%
0.8%
25
NAR Genomics and Bioinformatics
◐ 242 papers in training set
Top 4%
0.8%
26
Biology Methods and Protocols
◐ 61 papers in training set
Top 3%
0.6%
27
Patterns
○ 78 papers in training set
Top 3%
0.6%
28
Nucleic Acids Research
◐ 1281 papers in training set
Top 15%
0.6%
29
Neuroinformatics
○ 46 papers in training set
Top 1.0%
0.6%
30
Computer Methods and Programs in Biomedicine
○ 28 papers in training set
Top 1%
0.6%

"Similar papers" are the closest papers from that journal in the model's embedding space. They show what the match is built on, but the ranking comes mostly from a classifier over the whole training set, not from these examples alone.