Back

S3Mirror: Making Genomic Data Transfers Fast, Reliable, and Observable with DBOS

Vasquez-Grinnell, S.; Poliakov, A.

2025-06-16 bioinformatics
10.1101/2025.06.13.657723 bioRxiv
Show abstract

To meet the needs of a large pharmaceutical organization, we set out to create S3Mirror - an application for transferring large genomic sequencing datasets between S3 buckets quickly, reliably, and observably. We used the DBOS-Transact durable execution framework to achieve these goals and benchmarked the performance and cost of the application. S3Mirror is an open source DBOS Python application that can run in a variety of environments, including DBOS Cloud Pro where it runs as much as 40x faster than AWS DataSync at a fraction of the cost. Moreover, S3Mirror is resilient to failures and allows for real-time file-wise observability of ongoing and past transfers.

Matching journals

The top 4 journals account for 50% of the predicted probability mass.

1
GigaScience
◐ 212 papers in training set
Top 0.1%
19.3%
Similar papers in this journal
2
Bioinformatics
◐ 1204 papers in training set
Top 2%
13.0%
Similar papers in this journal
3
PLOS ONE
● 5266 papers in training set
Top 15%
12.4%
Similar papers in this journal
4
BMC Bioinformatics
○ 457 papers in training set
Top 1%
7.6%
Similar papers in this journal
50% of probability mass above
5
PLOS Computational Biology
● 1863 papers in training set
Top 8%
4.2%
Similar papers in this journal
6
Genome Biology
○ 637 papers in training set
Top 4%
2.8%
7
Bioinformatics Advances
◐ 203 papers in training set
Top 2%
2.5%
8
Nature Communications
○ 5641 papers in training set
Top 39%
2.5%
9
Scientific Reports
○ 3612 papers in training set
Top 44%
2.2%
10
Patterns
○ 78 papers in training set
Top 1%
2.0%
11
iScience
○ 1154 papers in training set
Top 16%
1.8%
12
Journal of Molecular Biology
○ 232 papers in training set
Top 2%
1.8%
13
SoftwareX
○ 15 papers in training set
Top 0.1%
1.8%
14
Gigabyte
● 62 papers in training set
Top 0.6%
1.8%
15
Journal of Open Source Software
● 25 papers in training set
Top 0.2%
1.8%
16
F1000Research
○ 88 papers in training set
Top 1%
1.8%
17
Frontiers in Genetics
○ 230 papers in training set
Top 3%
1.2%
18
Nature Methods
○ 385 papers in training set
Top 5%
1.2%
19
Journal of Bioinformatics and Systems Biology
○ 15 papers in training set
Top 0.3%
0.6%
20
PeerJ
◐ 308 papers in training set
Top 13%
0.5%
21
BMC Medical Genomics
○ 50 papers in training set
Top 2%
0.5%
22
PLOS Digital Health
● 106 papers in training set
Top 5%
0.5%
23
eLife
● 5828 papers in training set
Top 72%
0.5%
24
Computational and Structural Biotechnology Journal
● 242 papers in training set
Top 8%
0.5%
25
Journal of the American Medical Informatics Association
◐ 71 papers in training set
Top 2%
0.5%
26
BMC Research Notes
○ 33 papers in training set
Top 2%
0.5%
27
Journal of Medical Internet Research
◐ 87 papers in training set
Top 3%
0.5%

"Similar papers" are the closest papers from that journal in the model's embedding space. They show what the match is built on, but the ranking comes mostly from a classifier over the whole training set, not from these examples alone.