Morphology, Not Motion: Benchmarking Vision-Language Models on Multi-Sign Lung Ultrasound Interpretation
Lee, J.; Papastathopoulos-Katsaros, A.; Buralkin, I.; Shaik, Z.; Lee, B.; Leung, S. K.; Alavi, M.; Silva, B.; Choi, B.; Liu, Z.; Jeong, H.-H.
Show abstract
Background: Point-of-care lung ultrasound (LUS) interpretation requires specialized training. Vision-language models (VLMs) may automate multi-sign assessment but lack systematic clinical benchmarking. We benchmarked three LUS tasks on 125 evaluable cases drawn from 150 point-of-care ultrasound (POCUS) Atlas clips to establish baseline capabilities and isolate clinically relevant failure modes for VLM-based LUS interpretation. Methods: We evaluated a normalized multi-model benchmark on three tasks: pleural sliding detection (T1), lung-rocket (B-line) and consolidation classification (T2), and Posterolateral Alveolar and/or Pleural Syndrome (PLAPS; comprising posterolateral consolidation and pleural effusion) assessment (T3). Inputs were 10 uniformly sampled frames for all tasks, plus synthetic M-mode images for sliding. Results: We report four findings. First, M-mode improved T1 sliding over static frames for some models, but performance remained modest with wide uncertainty. Second, T2 pathology, identifiable from single frames, was discriminated most clearly above chance on a balanced cohort, with strong lung-rocket and anterior-consolidation F1 achieved by open-weight rather than closed-weight Claude models, although a significant between-model difference held only for lung rockets. Third, T3 PLAPS F1 was high for most models but reflected high positive-class prevalence rather than strong discrimination. Fourth, inter-model agreement was near chance (task-averaged Cohen's kappa = 0.02-0.29) and well below each model's self-consistency, indicating distinct, non-redundant error patterns. Conclusions: Static-frame VLMs produced discriminative labels for B-lines and anterior consolidation but remained unreliable for motion-dependent signs (pleural sliding) and prevalence-inflated outcomes (PLAPS). Synthetic M-mode partially recovered temporal information but did not achieve clinically sufficient sliding accuracy. These results support morphology-assisted use for B-lines and consolidation; motion-dependent signs and balanced endpoints require larger validation cohorts before clinical deployment
Matching journals
The top 8 journals account for 50% of the predicted probability mass.
Similar papers in this journal
- Generative AI Enables Medical Image Segmentation in Ultra Low-Data Regimes 93%
- Weakly supervised classification of rare aortic valve malformations using unlabeled cardiac MRI sequences 93%
- Segmenting functional tissue units across human organs using community-driven development of generalizable machine learning algorithms 92%
Similar papers in this journal
Similar papers in this journal
- Dual Adversarial Deconfounding Autoencoder for joint batch-effects removal from multi-center and multi-scanner radiomics data 93%
- Assisting Scalable Diagnosis Automatically via CT Images in the Combat against COVID-19 93%
- Aggregation of Cohorts for Histopathological Diagnosis with Deep Morphological Analysis 93%
Similar papers in this journal
- Deep learning models for COVID-19 chest x-ray classification: Preventing shortcut learning using feature disentanglement 94%
- BioFuse: An Embedding Fusion Framework for Biomedical Foundation Models 93%
- ai-corona : Radiologist-Assistant Deep Learning Framework for COVID-19 Diagnosis in Chest CT Scans 92%
"Similar papers" are the closest papers from that journal in the model's embedding space. They show what the match is built on, but the ranking comes mostly from a classifier over the whole training set, not from these examples alone.