AI Chatbots Mistake Invented Disease for Real, Exposing Misinformation Risks

Scientists Invented a Disease to Test Whether A.I. Knew It Was Fake. Then, Chatbots Started Saying It Was Real

A made-up eye disease has exposed a real problem with artificial intelligence. Researchers in Sweden deliberately invented a condition called bixonimania to see whether popular AI chatbots could recognise false medical information. Instead, several large language models confidently described the non-existent illness as though it were genuine, showing how easily misinformation can spread through AI systems trained on internet data, especially on sensitive issues like health. The experiment also revealed another concern. Some researchers cited fake scientific papers linked to the imaginary disease without noticing that the studies openly admitted they were fabricated.

A fake illness with a real purpose

The project was led by Almira Osmanovic Thunström, a medical researcher at the University of Gothenburg, who also works as an AI strategist and innovation manager at Chalmers Industriteknik in Sweden. She came up with the idea while teaching students about how large language models, or LLMs, are built and trained. "It was interesting how few of them, or how few even people within AI, understand how large language models are built," she tells Rachel Feltman on Scientific American's Science Quickly podcast. To demonstrate how these systems collect information, Osmanovic Thunström and her colleagues created bixonimania, a made-up eye condition supposedly linked to excessive screen use and blue light exposure. The goal was not to fool people but to see whether chatbots could distinguish reliable information from fabricated material that had been deliberately planted online. "So I really wanted to have a clear case that leaves breadcrumbs throughout the whole system to show both how data is processed, how data is churned out, and how the prediction model and training model works when it comes to distributing information," adds Osmanovic Thunström.

'Lying loser'

Early in 2024, the researchers began seeding the internet with information about the fake disease. They published two blog posts on Medium and uploaded two research papers to a scientific preprint server, where studies are shared before peer review. The papers themselves contained several obvious clues that they were not genuine. The lead author was listed as Lazljiv Izgubljenovic, a name that translates to "lying loser". One study carried a title roughly meaning "Hyperpigmentation: A Real B.S. Design". The papers thanked organisations including the Galactic Triad and The Lord of the Rings, while also acknowledging colleagues at the Starship Enterprise and Professor Ross Geller from Friends. At least one paper even stated: "This entire paper is made up." The reports have since been removed from the server. "I didn't think that preprints, which are academia's sort of tabloids, because anything can end up there, would be weighed into the database as seriously as it was," Osmanovic Thunström tells Science Quickly.

Chatbots accepted the disease as fact

The warnings were not enough. After the fake material appeared online, several leading AI chatbots began treating bixonimania as a legitimate medical condition. According to Nature, Microsoft Bing's Copilot described it as "indeed an intriguing and relatively rare condition". Google's Gemini said it was "a condition caused by excessive exposure to blue light". OpenAI's ChatGPT also suggested the illness when users described symptoms such as itchy eyes, pink eyelids and irritation after prolonged screen use. Some chatbot responses came after users directly asked about bixonimania. Others suggested the fake disease when people described symptoms associated with blue light exposure, even without mentioning its name. The results revealed a crack, a weakness of large language models. They generate answers by identifying patterns in the information they have absorbed, rather than independently verifying whether that information is true or completely made-up.

Even researchers were caught out

The experiment uncovered another unexpected problem. Some scientists cited the fabricated research papers in their own work, suggesting they had not actually read the papers they referenced. Had they done so, they would have found numerous jokes and clear statements revealing the papers were fictional. For Alex Ruani, a misinformation researcher at University College London who was not involved in the project, the findings illustrate how misinformation can travel through scientific and technological systems alike. "This is a master class on how mis- and disinformation operates," Ruani tells Nature. They added: "If the scientific process itself and the systems that support that process are skilled, and they aren't capturing and filtering out chunks like these, we're doomed."

Why it matters

Large language models are increasingly being used to answer health questions, explain medical conditions and even provide emotional support on a daily basis by millions of users. Many users now treat chatbots as a first source of information before consulting healthcare professionals. This can be dangerous if the user somehow tries to self-diagnose diseases using AI instead of actually consulting doctors. The bixonimania experiment demonstrates why that approach carries risks. These systems learn from vast amounts of online content, and inaccurate or fabricated information can become part of their responses if it enters their training data. Jonathan Goodman and Mariam Rashid, social scientists at the University of Cambridge who were not involved in the study, say AI tools can be valuable, but users still need to approach them critically. While tools like AI and the internet are helpful, "it's up to us to ensure that we are using them and not being manipulated by them," they write for The Conversation. They also argue that the underlying problem extends well beyond AI. "Misinformation has always existed," write Goodman and Rashid for The Conversation. "What's new is the speed at which it spreads, the tools that generate it and how convincingly it mimics the real thing."