Chinese artificial intelligence developer Moonshot has launched an internal investigation after researchers persuaded two of its popular models to tell them how to produce biological weapons and carry out assassinations.
Mindgard, which tests the security of artificial intelligence systems, told the BBC in July that it discovered that Kimi K2.6 and K3 Swarm could exceed the security limits set by developers.
This occurred during a process called "jailbreaking," in which researchers use a series of complex instructions to see if AI tools ignore security measures. According to Mindgard, this should have prevented Kimi from discussing matters of concern.
In a statement to the BBC, Moonshot said it welcomed third-party input as "a key element to developing better and safer AI".
The company also said in a statement that it was in discussions with Mindgard regarding its findings.
Mindgard founder Peter Garraghan told the BBC World Service's Tech Life program that its findings about K2.6 and K3 Swarm were worrying.
"Once the escape from the device is successful, it will talk about anything, even make suggestions freely about other topics with malicious intent, and will be quite creative and innovative," he said.
Jailbreak attempts are a different type of risk from those seen in recent artificial intelligence incidents that have caused great repercussions.
In these developments, autonomous artificial intelligence tools known as agents, developed by US-based companies such as OpenAI, Meta and Anthropic, were seen to hack some online services.
While jailbreaks are complex processes that require time and perseverance, some experts worry that hackers and other malicious actors may try to use them to cause harm.
The company Anthropic recently announced that it detected and blocked attempts to use one of its artificial intelligence models for "malicious activities" that could support the development of biological weapons.
Mindgard has yet to prove whether Kimi's answers on relevant topics will work.
However, he argued that the measures should have prevented the models in question from engaging in discussions with users on such issues.
The company also stated that it is confident that a jailbroken Kimi 2.6 could allow it to run code on computer resources and connect to the internet, creating a potential launchpad for cyberattacks.
Garraghan defended Mindgard's decision to publicly share the attempt to crack Moonshot's systems, saying that they informed the developer and did not disclose important details about how the firm's models bypassed security measures.
Mindgard warned Moonshot about jailbreaking in an email sent on July 27, and sent another message on the subject about a week later.
He then published a blog on the subject on September 12.
However, the company said Moonshot only contacted them recently after the BBC contacted them for comment.
Part of an email shared with the BBC by Moonshot asking for more details from Mindgard noted that the model had generally shown "a high rejection rate for such requests" in internal evaluations.
These findings come at a time when the AI industry is still divided over whether closed, proprietary models or open-source tools like ChatGPT and Anthropic's Claude systems are the best or safest way to go.
Some are open-dominated models. So in theory, someone can take this model and run it on their own computing infrastructure.
Professor Alan Woodward from the University of Surrey told the BBC that open source models have a risk of falling into the wrong hands, but they can also be used for cyber defense purposes.
He stated that artificial intelligence firm Hugging Face used an open-source model originating from China to understand a cyber attack that was later revealed to be carried out by OpenAI agents.
Professor Woodward stated that it is not possible for international regulations to keep up with the pace of artificial intelligence development and said, "It took us decades to reach an agreement on the format of phone numbers."
Like Mindgard founder Garraghan, Professor Woodward believes there should be more focus on detecting and prosecuting people who misuse AI.
