DINOSim: Zero-Shot Object Detection and Semantic Segmentation on Electron Microscopy Images
Gonzalez-Marfil, A.; Gomez-de-Mariscal, E.; Arganda-Carreras, I.
Show abstract
We present DINOSim, a novel method for detecting and segmenting objects in microscopy images without the need for large annotated datasets or additional training. DINOSim builds on the pretrained DINOv2 image encoder, which captures semantic information from images. By comparing the encoders features of images patches to those of a user-selected reference, DINOSim generates pseudo-labels that guide object detection and segmentation. Subsequently, a k-nearest neighbors framework is then used to refine predictions across new images. Our experiments show that DINOSim can effectively identify and segment previously unseen objects in diverse microscopy datasets, offering performance comparable to supervised approaches while avoiding the need for costly manual labeling. We also investigate how different choices of user prompts selection and model size affect accuracy and generalization. To make the method widely accessible, we provide an open-source Napari plugin (github.com/AAitorG/napari-DINOSim), enabling researchers to easily apply DINOSim to their own data. Overall, DINOSim offers a fast, flexible and practical solution for bioimage analysis, particularly valuable in resource-constrained settings.
Matching journals
The top 7 journals account for 50% of the predicted probability mass.
Similar papers in this journal
- RETINA: Reconstruction-based Pre-Trained Enhanced TransUNet for Electron Microscopy Segmentation on the CEM500K Dataset 97%
- A Deep Learning approach for time-consistent cell cycle phase prediction from microscopy data 97%
- Cell-TRACTR: A transformer-based model for end-to-end segmentation and tracking of cells 96%
Similar papers in this journal
- Generative AI Enables Medical Image Segmentation in Ultra Low-Data Regimes 95%
- Large-scale capture of hidden fluorescent labels for training generalizable markerless motion capture models 95%
- Segmenting functional tissue units across human organs using community-driven development of generalizable machine learning algorithms 94%
Similar papers in this journal
Similar papers in this journal
"Similar papers" are the closest papers from that journal in the model's embedding space. They show what the match is built on, but the ranking comes mostly from a classifier over the whole training set, not from these examples alone.