CT Multi-Task Learning with a Large Image-Text (LIT) Model
Niu, C.; Wang, G.
Show abstract
Large language models (LLM) not only empower multiple language tasks but also serve as a general interface across different spaces. Up to now, it has not been demonstrated yet how to effectively translate the successes of LLMs in the computer vision field to the medical imaging field which involves high-dimensional and multi-modal medical images. In this paper, we report a feasibility study of building a multi-task CT large image-text (LIT) model for lung cancer diagnosis by combining an LLM and a large image model (LIM). Specifically, the LLM and LIM are used as encoders to perceive multi-modal information under task-specific text prompts, which synergizes multi-source information and task-specific and patient-specific priors for optimized diagnostic performance. The key components of our LIT model and associated techniques are evaluated with an emphasis on 3D lung CT analysis. Our initial results show that the LIT model performs multiple medical tasks well, including lung segmentation, lung nodule detection, and lung cancer classification. Active efforts are in progress to develop large image-language models for superior medical imaging in diverse applications and optimal patient outcomes.
Matching journals
The top 8 journals account for 50% of the predicted probability mass.
Similar papers in this journal
Similar papers in this journal
- SN-FPN: Self-attention Nested Feature Pyramid Network for Digital Pathology Image Segmentation 93%
- Cell segmentation without annotation by unsupervised domain adaptation based on cooperative self-learning 93%
- Adaptive Frequency-Spatial Dual-Stream Network (AFS-DSN) for Nasal and Paranasal Sinus CT Segmentation 92%
Similar papers in this journal
- Deep learning models for COVID-19 chest x-ray classification: Preventing shortcut learning using feature disentanglement 95%
- BioFuse: An Embedding Fusion Framework for Biomedical Foundation Models 94%
- ai-corona : Radiologist-Assistant Deep Learning Framework for COVID-19 Diagnosis in Chest CT Scans 93%
Similar papers in this journal
- Generative AI Enables Medical Image Segmentation in Ultra Low-Data Regimes 94%
- Accurate recognition of colorectal cancer with semi-supervised deep learning on pathological images 94%
- Features fusion or not: harnessing multiple pathological foundation models using Meta-Encoder for downstream tasks fine-tuning 93%
"Similar papers" are the closest papers from that journal in the model's embedding space. They show what the match is built on, but the ranking comes mostly from a classifier over the whole training set, not from these examples alone.