A Multimodal Vision-text AI Copilot for Brain Disease Diagnosis and Medical Imaging

Integrating non-invasive brain imaging techniques, particularly computed tomography (CT) and magnetic resonance imaging (MRI), coupled with the advancement of artificial intelligence, is forging a key pathway for brain disease diagnosis, playing a vital role in safeguarding human health1-4. A robust artificial intelligence copilot is essential for clinical emergencies, functioning as the central processing unit for brain medical imaging systems, aiming to revolutionize the imaging process, expedite the diagnosis of diseases, and support treatment5-7. In this study, we developed an advanced multi-modal brain medical imaging foundational model named Brainfound, utilizing AI-generated content and image-text alignment technology, pre-trained on over 3 million brain CT images and over 7 million brain MRI images with their paired reports. As a clinical brain medical imaging multi-modal model, Brainfound achieved state of the art on seven downstream tasks, including brain disease diagnosis, brain lesion segmentation, MRI image enhancement, MRI cross-modality translation, automatic report generation, zero-shot brain disease classification, and free human-AI conversation. After thorough human-machine validation, Brainfound surpassed the current leading model by 51.75% in automatic report generation for brain imaging. In multiple-choice questions related to brain imaging, the accuracy of Brainfound outstripped GPT-4V by 47.68%, comparable to experienced doctors. We anticipate Brainfound, a clinical model with flexible visual and text input-output capabilities, will provide substantial support in brain medical imaging, clinical education, and human-in-the-loop medical diagnosis.