Back

SoftWipe - a tool and benchmark to assess scientific software quality

Zapletal, A.; Hoehler, D.; Sinz, C.; Stamatakis, A.

2020-10-13 bioinformatics
10.1101/2020.10.07.330621 bioRxiv
Show abstract

Scientific software from all areas of scientific research is pivotal to obtaining novel insights. Yet the quality of scientific software is rarely assessed, even though it might lead to incorrect scientific results in the worst case. Therefore, we have developed an open source tool and benchmark called SoftWipe, that provides a relative software quality ranking of 51 computational tools from diverse research areas. SoftWipe can be used in the review process of software papers and to inform the scientific software selection process.

Matching journals

The top 2 journals account for 50% of the predicted probability mass.

1
Bioinformatics
◐ 1204 papers in training set
Top 0.3%
38.8%
Similar papers in this journal
2
BMC Bioinformatics
○ 457 papers in training set
Top 0.4%
12.7%
Similar papers in this journal
50% of probability mass above
3
GigaScience
◐ 212 papers in training set
Top 0.6%
5.4%
Similar papers in this journal
4
Research Synthesis Methods
○ 20 papers in training set
Top 0.1%
4.0%
Similar papers in this journal
5
PeerJ
◐ 308 papers in training set
Top 2%
4.0%
Similar papers in this journal
6
PLOS ONE
● 5266 papers in training set
Top 37%
3.2%
7
Bioinformatics Advances
◐ 203 papers in training set
Top 2%
3.1%
8
Journal of Open Source Software
● 25 papers in training set
Top 0.1%
2.7%
9
eLife
● 5828 papers in training set
Top 46%
2.1%
10
Scientific Reports
○ 3612 papers in training set
Top 55%
1.7%
11
F1000Research
○ 88 papers in training set
Top 2%
1.7%
12
Nature Communications
○ 5641 papers in training set
Top 49%
1.3%
13
PLOS Biology
● 486 papers in training set
Top 10%
0.9%
14
Briefings in Bioinformatics
◐ 354 papers in training set
Top 7%
0.9%
15
BioData Mining
○ 22 papers in training set
Top 0.6%
0.9%
16
Computational and Structural Biotechnology Journal
● 242 papers in training set
Top 7%
0.8%
17
Journal of Bioinformatics and Systems Biology
○ 15 papers in training set
Top 0.3%
0.8%
18
iMeta
○ 10 papers in training set
Top 0.2%
0.8%
19
NAR Genomics and Bioinformatics
◐ 242 papers in training set
Top 4%
0.8%
20
Nature Methods
○ 385 papers in training set
Top 7%
0.8%
21
PLOS Computational Biology
● 1863 papers in training set
Top 20%
0.8%
22
Frontiers in Artificial Intelligence
○ 20 papers in training set
Top 0.8%
0.8%
23
Genome Biology
○ 637 papers in training set
Top 10%
0.6%
24
Frontiers in Behavioral Neuroscience
○ 49 papers in training set
Top 1%
0.6%
25
Epidemics
○ 116 papers in training set
Top 2%
0.6%

"Similar papers" are the closest papers from that journal in the model's embedding space. They show what the match is built on, but the ranking comes mostly from a classifier over the whole training set, not from these examples alone.