Back

Fast Loading and Transformation of Large-Volume Bioimaging Data Consistently Approaching the I/O Peak Speed

Cai, L.; Qu, X.; Zhou, H.; Li, N.; Gou, X.; Li, S.; Huang, J.; Guo, S.; Liu, X.; Lv, X.; Quan, T.; Zeng, S.

2025-12-12 bioinformatics
10.64898/2025.12.10.693603 bioRxiv
Show abstract

High-resolution large-volume biological imaging techniques are now widely used in biological research, but inefficiencies in data processing and visualization persist due to bottlenecks in loading/saving pipelines and limitations of conventional pyramid formats. To address these issues, we have decoupled data loading process into reading from disk and in-memory decompression, while data saving has been separated into in-memory compression and writing to disk. This approach allows us to dedicate more CPU cores to decompression and compression tasks, optimizing resource allocation and maximizing disk I/O throughput. As a result, both loading and saving processes operate near the disks peak data transfer capacity. Further, we integrated a video stream-inspired format into the pyramid structure, allowing direct access to regions of interest (ROIs) without loading extraneous data into RAM, reducing memory overhead. Compared to the latest methods, our approach improves pyramid data transformation efficiency by at least 6x and dramatically accelerates downstream tasks like visualization and deep learning inference. TeaserNovel bioimaging data processing pipeline achieves near-peak I/O throughput via decoupled architecture and video-formatting.

Matching journals

The top 5 journals account for 50% of the predicted probability mass.

1
Bioinformatics
◐ 1204 papers in training set
Top 1%
19.6%
Similar papers in this journal
2
Nature Methods
○ 385 papers in training set
Top 0.3%
19.6%
Similar papers in this journal
3
GigaScience
◐ 212 papers in training set
Top 0.6%
5.1%
Similar papers in this journal
4
Nature Communications
○ 5641 papers in training set
Top 28%
5.1%
Similar papers in this journal
5
Biological Imaging
◐ 15 papers in training set
Top 0.1%
3.7%
Similar papers in this journal
50% of probability mass above
6
Scientific Reports
○ 3612 papers in training set
Top 36%
2.8%
7
Communications Biology
○ 993 papers in training set
Top 8%
2.5%
8
PLOS Computational Biology
● 1863 papers in training set
Top 12%
2.2%
9
PLOS ONE
● 5266 papers in training set
Top 42%
2.2%
10
Patterns
○ 78 papers in training set
Top 0.9%
2.2%
11
BMC Bioinformatics
○ 457 papers in training set
Top 4%
1.8%
12
Advanced Science
○ 286 papers in training set
Top 4%
1.8%
13
Biomedical Optics Express
● 95 papers in training set
Top 0.5%
1.8%
14
Bioinformatics Advances
◐ 203 papers in training set
Top 3%
1.4%
15
Briefings in Bioinformatics
◐ 354 papers in training set
Top 5%
1.4%
16
Cell Reports Methods
○ 165 papers in training set
Top 2%
1.2%
17
NAR Genomics and Bioinformatics
◐ 242 papers in training set
Top 3%
1.2%
18
Journal of Microscopy
○ 20 papers in training set
Top 0.2%
1.2%
19
Small Methods
○ 29 papers in training set
Top 0.4%
1.2%
20
Genome Biology
○ 637 papers in training set
Top 8%
1.0%
21
Advanced Intelligent Systems
○ 11 papers in training set
Top 0.2%
1.0%
22
npj Imaging
○ 12 papers in training set
Top 0.2%
0.9%
23
eLife
● 5828 papers in training set
Top 68%
0.6%
24
Biophysical Journal
○ 631 papers in training set
Top 5%
0.6%
25
iScience
○ 1154 papers in training set
Top 39%
0.6%
26
Computational and Structural Biotechnology Journal
● 242 papers in training set
Top 7%
0.6%
27
Analytical Chemistry
● 218 papers in training set
Top 2%
0.6%
28
SoftwareX
○ 15 papers in training set
Top 0.4%
0.5%
29
Optics Letters
● 13 papers in training set
Top 0.2%
0.5%
30
Journal of Structural Biology
○ 64 papers in training set
Top 0.9%
0.5%

"Similar papers" are the closest papers from that journal in the model's embedding space. They show what the match is built on, but the ranking comes mostly from a classifier over the whole training set, not from these examples alone.