Back

ScrapPaper: A web scrapping method to extract journal information from PubMed and Google Scholar search result using Python.

Rafsanjani, M. R.

2022-03-09 bioinformatics
10.1101/2022.03.08.483427 bioRxiv
Show abstract

This paper introduces a program called ScrapPaper, a simple Python script that use web-scraping method to extract journal information from PubMed and Google Scholar search results page. Currently the motivation behind the program development is trying to solve a problem to obtain scientific literatures information especially the title and link and save as a list for further use such as in meta-analysis and comparative study of literatures. ScrapPaper advantage that it is simple to use with no prior programming experience and get the results ready within minutes (depending on the total search result). Web-scrapping is a very powerful method to extract information from the web and ScrapPaper employ several server friendly approaches accessing both PubMed and Google Scholar site. O_FIG_DISPLAY_L [Figure 1] M_FIG_DISPLAY C_FIG_DISPLAY

Matching journals

The top 5 journals account for 50% of the predicted probability mass.

1
PLOS ONE
● 5266 papers in training set
Top 6%
23.6%
Similar papers in this journal
2
F1000Research
○ 88 papers in training set
Top 0.1%
10.2%
Similar papers in this journal
3
BMC Bioinformatics
○ 457 papers in training set
Top 0.8%
8.3%
Similar papers in this journal
4
PeerJ
◐ 308 papers in training set
Top 0.4%
7.1%
Similar papers in this journal
5
Bioinformatics
◐ 1204 papers in training set
Top 5%
4.3%
Similar papers in this journal
50% of probability mass above
6
GigaScience
◐ 212 papers in training set
Top 1.0%
3.6%
7
Database
◐ 61 papers in training set
Top 0.3%
2.2%
8
Gigabyte
● 62 papers in training set
Top 0.4%
2.2%
9
Bioinformatics Advances
◐ 203 papers in training set
Top 3%
1.8%
10
Nature Protocols
○ 33 papers in training set
Top 0.3%
1.8%
11
Journal of Open Source Software
● 25 papers in training set
Top 0.2%
1.6%
12
Journal of Bioinformatics and Systems Biology
○ 15 papers in training set
Top 0.1%
1.6%
13
Scientific Reports
○ 3612 papers in training set
Top 59%
1.4%
14
Biology
○ 45 papers in training set
Top 0.3%
1.2%
15
SoftwareX
○ 15 papers in training set
Top 0.2%
1.2%
16
BMC Research Notes
○ 33 papers in training set
Top 0.6%
1.2%
17
Computational and Structural Biotechnology Journal
● 242 papers in training set
Top 4%
1.2%
18
Briefings in Bioinformatics
◐ 354 papers in training set
Top 5%
1.2%
19
DIGITAL HEALTH
○ 17 papers in training set
Top 0.6%
1.1%
20
JMIR mHealth and uHealth
◐ 11 papers in training set
Top 0.2%
1.0%
21
STAR Protocols
○ 18 papers in training set
Top 0.2%
1.0%
22
Journal of Medical Internet Research
◐ 87 papers in training set
Top 2%
0.9%
23
BioData Mining
○ 22 papers in training set
Top 0.7%
0.6%
24
Frontiers in Behavioral Neuroscience
○ 49 papers in training set
Top 1.0%
0.6%
25
PLOS Computational Biology
● 1863 papers in training set
Top 21%
0.6%
26
Journal of the American Medical Informatics Association
◐ 71 papers in training set
Top 2%
0.6%
27
Frontiers in Artificial Intelligence
○ 20 papers in training set
Top 0.8%
0.6%
28
Research Synthesis Methods
○ 20 papers in training set
Top 0.2%
0.6%
29
Genomics, Proteomics & Bioinformatics
188 papers in training set
Top 2%
0.6%
30
G3 Genes|Genomes|Genetics
◐ 351 papers in training set
Top 4%
0.6%

"Similar papers" are the closest papers from that journal in the model's embedding space. They show what the match is built on, but the ranking comes mostly from a classifier over the whole training set, not from these examples alone.