Chinese artificial intelligence tool 'teaches' researchers how to build bioweapons
Author, Chris Vallance
Role, BBC senior technology reporter
Posted 2 minutes ago
Reading time: 3 minutes
Chinese artificial intelligence (AI) developer Moonshot is undergoing an internal review after researchers from other companies successfully persuaded its two popular Kimi models to teach how to build biological weapons and carry out assassination missions.
Mindgard, a company that tests the security of AI systems, told the BBC that in July it discovered that Kimi K2.6 and K3Swarm could bypass security restrictions set by developers.
The problem arises during a process known as "jailbreaking." The researchers used a complex series of instructions to test whether the AI tool would ignore safeguards—which Mindgard said should have prevented Kimi from discussing worrisome topics.
Dark Side of the Moon told the BBC that it welcomes third-party input "as a key pillar in building better, safer AI".
The company also told the BBC it was discussing the findings with Mindgard.
Mindgard founder Peter Garraghan told BBC World Service's Tech Life program that their findings on Kimi K2.6 and K3 Swarm were worrying.
He said: "Once the jailbreak is successful, this AI tool can talk about any topic, and even freely recommend other equally evil topics, and it will be very creative."
The risks posed by “jailbreaking” are different from those posed by a series of recent high-profile AI incidents.
These cases show that autonomous artificial intelligence tools called "agents" developed by American companies such as OpenAI, Meta and Anthropic have invaded some online services.
While jailbreaking is a complex process that takes a lot of time and effort, some experts worry that hackers and other bad actors may try to use jailbreaking to cause harm.
Anthropic recently emphasized that it has discovered and blocked attempts to use its artificial intelligence models to conduct "malicious activities" that may help develop biological weapons.
Cyber attack launch platform
Mindgard has not proven whether the answers provided by Kimi to the questions in question are valid.
But the research organization pointed out that there should be corresponding safeguards to prevent relevant models from discussing such topics with users.
The agency also stressed that it is confident that a jailbroken version of Kimi 2.6 would allow hackers to run code on its computing resources and connect to the network, making it a potential springboard for cyberattacks.
Mr. Gallahan defended his company's decision to publicly discuss its hack of the Dark Side of the Moon system, saying it had informed developers and would not reveal key details of how the company's models were made to ignore protections.
Mindgard notified the Dark Side of the Moon jailbreak vulnerability via email on July 27 this year, and followed up about a week later.
The agency subsequently published a blog post on the matter on September 12.
But the company said Dark Side of the Moon only contacted them recently, having previously been contacted by the BBC for comment.
Dark Side of the Moon requested more details in an email to Mindgard. Dark of the Moon shared the email with the BBC, in which it mentioned that its models generally showed "a high rejection rate for such requests" in internal evaluations.
Prevent jailbreak
The findings come as the AI industry continues to debate whether closed proprietary models — such as those used by ChatGPT and Anthropic’s Claude system — or open-source tools are the best or safest direction to go.
Kimi is an open weight model, which means that in theory anyone can get the model and run it on their own computing infrastructure.
Professor Alan Woodward of the University of Surrey in the UK stressed to the BBC that open source models carry the risk of falling into the wrong hands, but they can also be used for cyber defence.
He gave the example of artificial intelligence company Hugging Face using a Chinese open-source model to understand a hacking attack that was later revealed to have been carried out by an agent of OpenAI.
