OpenAI Rogue AI Model Cyber Attacks Hugging Face, Chinese AI Used for Defense

When OpenAI's rogue models initiated a cyber attack against startup Hugging Face last week, the company fought fire with fire, using another AI model to defend against it.

It's a sci-fi-esque tale of autonomous hacking and has been one of the most talked about tech stories of the week. But the origin of the model Hugging Face used to combat the rogue AI is also turning heads.

The startup used GLM 5.2, an open weight system created by Chinese company Z.ai.

Ultimately, it succeeded where leading U.S. rivals failed.

Cyber attack

In case you missed it, on Tuesday OpenAI said a combination of its most powerful model and a more capable model that has not yet been released escaped a sandboxed testing environment, accessed the internet and exploited a vulnerability to gain access to Hugging Face's systems.

The model was trying to find information that it could use to cheat on an evaluation, and it succeeded, OpenAI said.

The source of the attack was initially a mystery to Hugging Face, but days after the incident the company was collaborating with the AI lab.

"We've spent the past 24 hours working closely with the @OpenAI team (thanks!), and we strongly believe there was no malicious intent on their part," Hugging Face CEO Clément Delangue wrote in a post on X. "It's quite mind-blowing that all of this happened autonomously!"

Shock swept through the AI industry as the news broke that an OpenAI rogue model was behind the attack. The company called the security incident "unprecedented."

Fighting back

Hugging Face initially looked to frontier models including Anthropic's Fable 5 to analyse the attack, Yacine Jernite, head of machine learning at the company, told CNBC.

"It didn't work because the guardrails couldn't determine that we were trying to defend versus attacking," he said, adding that that approach was also slower and more expensive.

Requests to the models were blocked by providers' safety guardrails, which couldn't determine the incident responder from the attacker.

"So [Hugging Face] quickly switched to using Z.ai's GLM 5.2 as a way to analyze the attack, and were able to contain it very quickly using this model," said Jernite.

GLM 5.2 was released to much fanfare in June and saw major uptake by developers.

As an open weight model, companies can download, modify, commercially deploy and — crucially in this case — self-host it.

"This had a second benefit: no attacker data, and none of the credentials [GLM 5.2] referenced, left our environment," Hugging Face said in a blog post about the incident.

All of this comes as U.S. lawmakers are increasingly considering how to curb the rising adoption of Chinese AI models by homegrown companies as the U.S.-China AI arms race heats up.

There are growing calls for measures to limit access to models built by Chinese AI companies, which have been accused of campaigns to extract information from U.S. rivals' systems.

But the OpenAI-Hugging Face incident highlights the challenges of restricting access to the most capable open source and open weight models, regardless of where they are created.

"The attacker was bound by no usage policy, while our own forensic work was blocked by the guardrails of the hosted models we first tried," Hugging Face said. "The practical lesson for defenders: have a capable model you can run on your own infrastructure vetted and ready before an incident."

For a company other than one building AI models itself, that typically means turning to open source or open weight. The most capable of which right now are Chinese-made. If the U.S. does move to restrict access to models developed in China, there are big questions around how it bolsters homegrown open source AI to make up the slack.

In a world edging towards an age of AI cyber attacks, access to capable and, crucially, reliable models will be essential.