Board-Level Performance of Leading Open-Weight Vision-Language Models on the Japanese Diagnostic Radiology Board Examination: Reasoning, Image-Input, and Language Effects

Purpose: To evaluate the latest open-weight vision-language models (VLMs) on the Japanese Diagnostic Radiology Board Examination (JDRBE), assessing overall accuracy and the effects of image input, reasoning, and language. Materials and Methods: In this retrospective study, 29 open-weight VLMs from 13 developers, released in or after January 2025, were evaluated on 327 image-bearing questions from four years of the JDRBE, a non-public benchmark with low risk of data leakage. Each question was answered by each model with and without the image(s), under three language conditions and with reasoning enabled and disabled. Accuracy was the primary outcome, and within-model differences were tested with paired bootstrap confidence intervals and sign-flip permutation tests with Benjamini-Hochberg correction. Results: In the Japanese condition with image input and reasoning, the leading models reached 73.7% (gemma-4-31B-it), 73.1% (Qwen3.5-397B-A17B), and 72.1% (Kimi-K2.6). On the 2025 subset, these three models (74.1%-75.5%) scored above the mean accuracy of five newly board-certified radiologists who passed the 2025 examination (72%; range, 65%-83%). Accuracy broadly scaled with model size, although compact gemma-4-31B-it matched larger models. Enabling reasoning improved accuracy in nearly all models and the contribution of image input was larger when reasoning was enabled, particularly in higher-performing models. English prompts generally outperformed Japanese prompts. Conclusion: Several open-weight VLMs, without medical adaptation, performed at or above the mean of newly board-certified radiologists on the JDRBE, with both model size and reasoning contributing. The highest Japanese-language accuracy came from a compact model suitable for parameter-efficient fine-tuning and serving on a single graphics processing unit.