Anthropic said it has blocked several attempts this year by scientists to use its leading artificial intelligence models to conduct research that could help develop bioweapons.
In a report describing the misuse of its AI models, Anthropic said it could not determine whether the research was for legitimate or nefarious purposes, because valid biological research — the kind that leads to breakthroughs such as vaccines — could also help engineer dangerous pathogens. Faced with this uncertainty, Anthropic said it is choosing to err on the side of caution because the consequences of missing malicious activity could be severe.
"You're not going to see someone like in a comic book saying, 'Hey, I want to make a bioweapon to kill everyone,'" Jacob Klein, director of threat intelligence at Anthropic, said in an interview. "The situation is extremely delicate."
The potential for cutting-edge AI models to facilitate the development of known or entirely new biological pathogens is one of the most serious concerns experts have about a technology that is advancing so rapidly that even its leading designers doubt humans can fully control it.
Biological abuse has received less attention as an existential risk for AI than catastrophic cyberattacks or AI agents inconsistent with human intent, in part because past cases have typically emerged only in research settings, not the real world.
Andrew Weber, senior fellow at the Strategic Risk Council, reviewed Anthropic's report before its release. He said the findings were a "chilling example of the rapid advancement of state-backed bioweapons developers' ability to leverage leading AI models."
Anthropic's lengthy report released Thursday lists a series of instances of abuse of various models of its Claude chatbot over the past eight months. Some of the examples are similar to past disclosures by Anthropic and other AI labs, including surveillance of dissident and diaspora communities by actors suspected of being linked to the Chinese and Iranian governments.
The report also alleges that Russian state media used Claude to generate online propaganda disguised as independent reporting, including fabricated content about the Moldovan elections.
Anthropic documented what it considers to be a new type of abuse: attempts to exploit Claude to develop software for the design and development of conventional weapons, including guns, missiles, armed drones and bombs. The report details three cases in China, two in Russia and one in Yemen.
The report did not name which party was involved in the case in Yemen, but the context made it clear it was referring to the Iran-backed Houthi rebels. The use of AI by terrorist networks is a growing concern for U.S. security officials.
But of all the threat categories listed in the report, biological research may be the most worrisome. Anthropic did not disclose the names of the researchers or institutions it blocked, their countries or the specific biological agents involved, in part because it was unsure of their purposes.
Still, Anthropic said the scientists circumvented controls designed to prevent users in blocked areas from accessing their AI models and managed to "obfuscate the purpose of the research to circumvent our security guardrails." The company banned the accounts in question.
In one case in May, a scientist asked Claude for help writing a grant application to fund research into the chikungunya virus. This is a mosquito-borne disease that can cause severe pain and other symptoms for months. Such research, known as gain-of-function research, may be legal and lead to vaccine development, but it may also create super viruses. In this case, scientists wanted to engineer mutations in the virus that would make it more harmful when repeatedly infecting live animals.
Anthropic said it found the study concerning, in part because it was able to discern that the work was intended to be performed at a military research institution.
"What we don't know is whether there was a weaponization intent in this study," Klein said. "But it's concerning for military agencies to conduct gain-of-function studies."
Anthropic said a review last year of its old models showed they were not yet effective at assisting in dangerous biological research. The company says current models are more capable of completing complex scientific research, prompting it to tighten safety guardrails to limit questions asked in research on multiple dual-use organisms.
