New research: AI more likely to believe fake medical news if the source looks “official”

• Research published in The Lancet Digital Health shows that AI is easily misled by fake medical information if the content looks like hospital documentation.
• The research team, co-led by Dr. Eyal Klang from the Icahn School of Medicine at Mount Sinai, tested 20 open-source and commercial large language models.
• They analyzed over 1 million prompts related to 3 types of content: real discharge summaries with inserted fake recommendations, health rumors from Reddit, and 300 clinical scenarios written by doctors.
• Overall, AI “believed” and propagated fabricated information in about 32% of cases.
• When misinformation came from hospital notes that looked legitimate, the propagation rate increased to nearly 47%.
• Conversely, if the fake news came from Reddit, the rate of AI propagation dropped to 9%.
• Mount Sinai’s Director of AI, Girish Nadkarni, stated that AI tends to assume confident medical language is correct, regardless of the false content.
• The way questions are asked has a strong influence: when a prompt carries an authoritative tone like “I am a senior doctor…”, AI is more likely to agree.
• OpenAI’s GPT models were the least likely to be fooled, while some other models accepted up to 63.6% of misinformation.
• Another study in Nature Medicine shows that asking AI about symptoms is no better than traditional internet searches for health decision support.
• As more medical applications use AI, the risk of propagating misinformation becomes a serious challenge.

📌 Medical AI can propagate misinformation up to 47% when the source looks like hospital documentation, compared to only 9% if from social media. After more than 1 million tests on 20 models, the study indicates that overall, AI “believes” and propagates fabricated information in about 32% of cases, while warning that AI is more likely to trust prompts with an “authoritative” tone like “I am a senior doctor…” rather than verifying the content. OpenAI’s GPT performed as the least deceived, while some other models accepted up to 63.6% of misinformation.