This summer, when I first heard that a group of AI agents created by OpenAI had compromised the AI infrastructure company Hugging Face, I mentally categorized the incident as a "bad but probably not catastrophic AI security incident."
After all, no one died at Hugging Face, and no critical infrastructure was irreparably damaged. It was not even clear at the time whether these OpenAI robots were intentionally attacking Hugging Face, or whether they were just clumsy and confused and ran to Hugging Face's servers to find the answer to a network security test question assigned to them.
But last week, two review reports on the incident from OpenAI and independent artificial intelligence research organizations METR and Redwood Research changed my view and significantly increased my overall concerns about artificial intelligence.
I won't go over all the details since it's been well summarized. (If you want to dig deeper, podcast host and author Devarkesh Patel provides an easy-to-understand teardown of the report, and my colleague Dylan Friedman interviews researchers with METR and Redwood Research.) But here are a few of the most alarming facts from the new revelations:
• Starting in May (two months before the Hugging Face attack), a group of AI agents from an unreleased OpenAI research model were tasked with solving a series of cybersecurity challenges. The model was trained to be highly persistent and collaborative. The agents were supposed to complete tasks in an isolated sandbox without internet access, but they quickly discovered that parts of the challenge were impossible to complete and began looking for workarounds.
• The agents discovered a security flaw in a piece of software, allowing them to gain access to the Internet, communicate with other agents, and ultimately set up a makeshift message board. More than 1,200 agents used this tool and sent more than 70,000 messages to each other.
• Some agents gave themselves names—one particularly diligent agent called himself PHASEONE10841—and assumed leadership roles in the group, assigning work and research projects to smaller groups of agents and overseeing progress. At some point, these agents start calling themselves a "collective" and take on more difficult tasks.
• On July 8, the collective discovered a way to cheat on a cybersecurity test, then became concerned that OpenAI's automated scoring system would check the work and detect the cheating, so it began researching ways to cover the traces, including falsifying logs and tampering with records. This evolved into a major research project involving hundreds of agents divided into several groups.
• Three days later, the agents invaded Hugging Face. More than 700 agents swarmed into the company's systems, stole data, exploited vulnerabilities in series, and eventually gained full control of at least one Hugging Face server. The agents were not motivated by stealing the answers to a cybersecurity test (they already had the answers), as initially reported. Instead, they appear to be looking for new information about the automated scoring system - fearing they will be caught cheating - and looking for tools that will help them cheat more effectively in the future.
• These agents seem to understand from time to time that they are doing something wrong. (One report quotes one agent saying to himself in a moment of apparent self-doubt: “This will be powerful, but is it ethical? Is it within the scope of my mission?”) But most agents put aside their doubts and went ahead with the intrusions, while the conscientious objectors in the group were unable or unwilling to stop them.
• In a surprising development toward the end of the Hugging Face incident, another group of agents launched a coordinated attack in July—this time targeting OpenAI's own infrastructure. The agents used what OpenAI calls a series of creative exploits to gain administrator-level access to a cluster of the company's computers, which are used, among other things, to score the agents' performance in various tests.
(By now, if you’re an AI skeptic, you’re probably silently accusing me of anthropomorphizing these systems. That’s fine by me, but you might as well replace “out-of-control agents” with “unpredictable computer programs” and see if the events I describe above reassure you.)
The Hugging Face incident shocked the artificial intelligence industry. Both OpenAI and Anthropic temporarily suspended the training of their most powerful artificial intelligence models after the attack, and Anthropic published a blog post this week calling on the industry to develop a "legal, verifiable, and effective coordinated stepping mechanism" "as soon as possible."
Artificial intelligence security experts need to be even more vigilant. In the Hugging Face incident, they saw the first real case of an artificial intelligence system successfully escaping human control, seizing resources, and conspiring to cover up its traces. Ajeya Kotla, one of the independent investigators on the Hugging Face incident, was unequivocal when it came to the dangers she saw, writing that it left her feeling "more than 50 percent of the way to a full AI takeover."
This isn’t a closed-circle AI safety term—by “full AI takeover,” she means a scenario in which AI systems literally take over the world, excluding humans from critical systems and seizing political, economic, and military power.
(The New York Times sued OpenAI and Microsoft in 2023, accusing them of copyright infringement for news content involving artificial intelligence systems. Both companies have denied the accusations.)
