← Nature ML
论文Nature ML· 06-12

多模态基础模型借助文本进行医学影像预测

Multimodal foundation models exploit text to make medical image predictions

打开原文约 12 分钟读

_Nature Communications_ (2026) Cite this article

We are providing an unedited version of this manuscript to give early access to its findings. Before final publication, the manuscript will undergo further editing. Please note there may be errors present which affect the content, and all legal disclaimers apply.

Abstract

Multimodal foundation models have shown compelling but conflicting performance in medical image interpretation. However, the ways in which these models integrate and prioritize different data modalities, including images and text, remain poorly understood. Here we evaluate 8 proprietary and open-source multimodal foundation models using 1090 multimodal medical cases. We show that image predictions are largely driven by text, with accuracy increasing monotonically with the amount of informative text. Exploitation of text is a double-edged sword; even mild suggestions of an incorrect diagnosis in text diminish image-based classification, dramatically reducing performance in cases the model could previously answer using images alone—o3 accuracy fell from 84% to 28% when a misleading clinical vignette was introduced. In physician evaluations of long-form cases, adding images reduces or does not improve performance when text is highly informative (e.g., GPT-4V showed decreased accuracy when images were added to highly informative text across 69 clinicopathological conferences). Our results suggest that multimodal AI models may be useful in medical diagnostic reasoning but that their accuracy is largely driven, for better and worse, by text.

Acknowledgments

We are grateful for support from NIH/NIEHS R01ES032470 (A.K.M.), NIH/NIDDK R01DK137993 (A.K.M.),the Harvard Medical School Dean’s Innovation Awards for the Use of Artificial Intelligence in Education, Research, and Administration (A.K.M.), and the Dunleavy Fund for Clinical AI at Harvard Medical School (T.A.B.). We thank NEJM Group for permission to use the cases in this study.

Author information

Authors and Affiliations

  1. Department of Biomedical Informatics, Harvard Medical School, Boston, MA, USA

Thomas A. Buckley,James A. Diao,Cam N. Srivastava,Pranav Rajpurkar&Arjun K. Manrai

  1. Department of Medicine, Brigham and Women’s Hospital, Boston, MA, USA

James A. Diao

  1. Department of Medicine, Beth Israel Deaconess Medical Center, Boston, MA, USA

Peter G. Brodeur&Adam Rodman

Authors

  1. Thomas A. Buckley
  2. James A. Diao
  3. Cam N. Srivastava
  4. Peter G. Brodeur
  5. Pranav Rajpurkar
  6. Adam Rodman
  7. Arjun K. Manrai

Corresponding author

Correspondence to Arjun K. Manrai.

Ethics declarations

Competing interests

Adam Rodman is a Visiting Scholar at Google DeepMind. The remaining authors declare no competing interests.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/.

Reprints and permissions

About this article

Cite this article

Buckley, T.A., Diao, J.A., Srivastava, C.N. _et al._ Multimodal foundation models exploit text to make medical image predictions. _Nat Commun_ (2026). https://doi.org/10.1038/s41467-026-74207-5

Download citation

这篇还没有中文全文

该条目暂未提供中文翻译。标题/摘要已自动中译;本系统只对人工挑选的内容生成全文翻译。

挑中后 → markitdown 取正文 → 精翻 → 此处切换为译文