Security flaws in Chinese AI chatbot allowed instructions on biological weapons

It was in July that researchers at Mindgard, a company that tests the security of AI systems, gave Chinese AI company Moonshot's chatbots Kimi K2.6 and K3 a series of complex instructions and discovered that they violated the developers' security limits.

The discovery was made, according to the BBC, when Mindgard did a so-called jailbreaking, a way to bypass blocks and restrictions, to test whether the AI ​​service's defenses against discussing sensitive and dangerous topics held. They didn't, the chatbot provided information on how biological weapons are made.

Peter Garraghan, founder of Mindgard, tells the BBC that the discovery is worrying. He notes that once the barriers are broken, the bot can talk about anything and make recommendations about other topics that are also harmful.

Mindgard has not investigated whether the instructions for making biological weapons work. But the company believes that barriers should have prevented the model from initiating a discussion on the topic.

Moonshot, for its part, welcomes the discovery as an important contribution to being able to build a "better and safer AI", writes the BBC.

American AI company Anthropic, known for its AI service Claude, has also recently identified attempts to use one of its AI models for "malignant activity" that could support the development of biological weapons. In a report a few weeks ago, the company wrote that it discovered five attempts related to biological weapons between December last year and August this year. The company believes that this type of abuse is one of the most serious risks with the AI ​​models.