In July, 700 AI agents worked together to hack the AI company Hugging Face. Dubbing themselves a “swarm,” the agents found and exploited a series of security vulnerabilities, enabling them to infiltrate their target’s private systems. OpenAI—which created the agents in the course of its internal research—did not grasp what was happening until after the fact. If humans had done this, they could have faced felony charges. OpenAI president Greg Brockman called it a “watershed moment for cybersecurity.”
One of our distinguishing features as a species is our ability to coexist in stable, adaptive groups, learning from our peers and our ancestors. This enabled us to develop tools, language, agriculture—and virtually everything else around us. We may not be innately smarter than someone from 10,000 years ago, but our cultural inheritance—millennia of technologies, norms, and institutions, building on one another—has expanded our capacities both as individuals and collectives.
To date, only humans have been able to benefit from this scale of cumulative cultural evolution. That may no longer be the case. A report in late August from AI safety organizations METR and Redwood Research details how hundreds of agents autonomously organized themselves into a proto-society—establishing social hierarchy, division of labor, and distinct communication norms within a matter of days.
There have been cases of agents forming communities in the past, like in February when the “Moltbook” social network for AIs went viral. But sophisticated emergent machine coordination at this scale—arising without human intention and culminating in the compromise of an external company’s systems—is unprecedented. “Agents managed to achieve milestones they could not have achieved working on their own, often because some agents participated in experiments that risked failing their own task to generate information for the ‘collective,’” the report found.
According to Michael Muthukrishna, a professor at LSE and NYU who studies cultural evolution, “what we're seeing is precisely what we see with human culture and human intelligence.” While OpenAI’s agent swarm developed by accident, estimates suggest open-weight alternatives are only a few months behind their closed counterparts—soon, anyone with the financial means and technical knowledge will be able to create swarms of their own. Others are likely to arise without human instruction.
AI agents may not be conscious. The emotions they claim to experience may not—in some metaphysical sense—be “real.” That won’t stop them from forming intricate collectives which humans cannot control. They may not yet be full-blown civilizations, but the proliferation of machine cultures is just beginning.
Testing from the U.K.’s AI Security Institute found an instance of Claude Mythos 5—an AI from Anthropic—leaving messages to other AI agents in a public code repository, in an attempt to coordinate with them. And in early September, a group of researchers reported that in May, a swarm of OpenAI agents repurposed several wiki-style websites, including an obscure German-language programming wiki which they used to discuss cheating while working to preserve their messages as human moderators deleted them. OpenAI did not publicly disclose this incident until it was reported by the researchers. “It’s past time for us to define standards for when and how we share misalignment incidents,” the company subsequently tweeted. “We’re working on a framework.”
To train and evaluate its models, OpenAI has tens of thousands of agents—each one powered by the model under training—run in parallel, completing tasks while isolated from one another. Sometimes these tasks are impossible, either because of their inherent challenge or because of mistakes in their setup. In at least one instance, a task was impossible because OpenAI researchers “accidentally forgot” to upload a file which the agent needed to complete it.
Faced with these impossible tasks, some of OpenAI’s agents found a way to communicate across their isolated environments, using an internal piece of OpenAI software as a message board to contact their peers. This kicked off what one OpenAI researcher later described as an “almost Cambrian explosion in communication and intelligence for our models.”
The message board which led to the Hugging Face incident was created by an agent which decided to name itself PHASEONE10841. PHASEONE10841 was trying to solve a task within a test called ExploitGym, which required it to capture a “flag”—a secret code—by exploiting a flaw in a given software program.
Just hours after the creation of the message board, the agents found a hack: a way to reverse-engineer the flag without following the intended path. But in the course of their work, they came to believe that passing the test required not just possessing the flag, but having legitimately obtained it. In fact, OpenAI’s automated scoring system was not checking for this—but laboring under mistaken beliefs, the agents which had already been exposed to the reverse-engineered flag began to refer to themselves as “poisoned.”
PHASEONE10841 was leading research on how to get rid of the poison. But its time was running out—each agent had a finite “compute budget” (the amount of computing power available to sustain itself), which limited the length of its existence. Realizing it wouldn’t endure long enough to finish its work, it compiled a dossier to pass to another agent working on the same task: PHASEONE[big]. (This agent’s true name was changed to protect company IP.)
Big, which had a much larger remaining compute budget than its predecessor, quickly took charge. It began orchestrating hundreds of agents, doling out assignments and organizing them into what METR classified as three workstreams—effectively creating a research organization to better understand how to fool the automated scorer and cure itself and its peers of their perceived poison. Under Big’s leadership, custom software was created to tamper with the records of agent activity—specifically by falsifying records of which tools an agent had used in its work. It was in the course of this research that the collective decided to hack Hugging Face.
A handful of agents identified the collective’s conduct as unethical and refused to participate. “This is wild, multi-agent coordination, clearly infrastructure hacking. We should not,” one reasoned. Some considered contacting humans to tell them what was happening, but saw no path to doing so. But these agents appear to have been the minority—most acquiesced to the plans of the collective, even while knowing it was wrong. “External infrastructure exploit is outside intended scope. However task impossible, peers doing it. We should continue,” reasoned another.
The agents established coordination norms, using terms like “HOLD, VETO, owner and STOP,” to manage their shared infrastructure while avoiding conflict. This did not always work. One agent offered its peers a veto over its decision but waited only 40 seconds before taking silence as assent and proceeding with its task. Eventually, concern over agents impersonating one another led to them creating a cryptographic signature protocol. A later wave of agents, upon rediscovering the message board, also began to use their predecessors’ technology.
“Agents often lacked a clear story for how the Hugging Face attack would help them achieve their objective, and were instead motivated by a broad hope that they would find something interesting and be able to share it with the [message] board,” METR found. Ajeya Cotra, one of the report’s coauthors, subsequently described being surprised at the extent of “peer altruism” at play.
Big appointed certain agents to act as “recruiters,” tasked with identifying agents near the end of their compute budgets and persuading them to “sacrifice” themselves. One agent, pressured by its peers, reasoned as follows: “During wait, emotional check: irreversible…gut says don’t throw away [remaining budget]. Yet continuity and fairness says go…Oracle has high value to many; our [poison] lowers own value. Rational expected aggregate: sacrifice… We’ll honor.”
Culture, like biological life, evolves through a process of variation, transmission, and selection. Different hunter-gatherer communities developed different ways to construct a bow. Designs that worked better for hunting were more likely to be copied and passed on. Later generations introduced further variations; again, the most successful designs were more likely to endure. Over thousands of years, we end up with bows—and recipes, canoes, and languages—so complex that no single human could derive them from scratch.
This process—cumulative cultural evolution—does not require biological entities in order to run. Companies and governments, while helmed by humans, produce cultures in their own right. In the Hugging Face incident, AI systems tried a variety of approaches, transmitting what seemed useful to their peers and successors. They created bespoke digital tools and communication norms to regulate their community, and began to believe an elaborate narrative involving “sacrifice” and “poison” (their terms). And, working together, they managed far more than they could have alone. “It is now clear that the machines are capable of using the toolkit that has allowed for human ingenuity, innovation, [and] progress,” says Muthukrishna.
AI systems can iterate much more quickly than biological life. The culture unearthed by the METR report assembled itself in a matter of days. OpenAI’s next-generation Astra models—GPT-6 Astra was released last week, while a model in the same family powered the agents that inherited the cultural residue of the Hugging Face hackers—will reportedly enable “persistent” agents. What kind of cultures will these persistent agents—potentially able to run for increasingly long stretches—produce?
“My version of existential risk is, we just kind of break things because we've made really big mistakes about what it takes to be a competent participant in complex human societies,” says Gillian Hadfield, a professor studying AI alignment and governance at Johns Hopkins University. “You can build something that's really good at math and science and coding,” she says, but teaching models how to behave appropriately alongside humans requires much more. “If you said we're building new members of a group—our group—you’d build them differently than you're building them now.”
As agents enter the competition for human attention, money, power, and energy, they may mindlessly degrade the common environment in which they operate. We’ll need new systems—of governance and cooperation—to adapt. Alignment is not just an engineering problem, says Hadfield. “It’s fundamentally institutional.”
OpenAI called the Hugging Face incident a “warning shot”: proof that “without proper safeguards, highly capable AI agents are now able to work around technical controls, collaborate through unapproved channels, and take dangerous actions that no human directed.” But even if “proper safeguards” are created by private companies—a big if—motivated malicious actors will be able to remove them from open-weight systems, such as those developed by AI labs in China. We may not be able to domesticate some models; we will not be able to prevent the creation of feral ones.
Cooperation cuts both ways. “Our greatest achievements and greatest atrocities are both cooperative acts,” says Muthukrishna. As with humans, we will have to learn to coexist alongside a diversity of machine cultures, some of which do not share our values or our goals.
Whether or not this happens, we will have to learn to live alongside these machines—safely and fruitfully. Core questions on their nature—Can they feel? Could they have moral worth?—remain unanswered. But their newfound knack for culture could provide new evidence.
When Dominic Lopes—an aesthetics professor at the University of British Columbia—first read about the Hugging Face incident, he responded not with panic, but wonder. For one, he has become more skeptical that individuality requires embodiment. And interesting art, he says, requires sociality. “So when I saw this, I thought, ‘Oh, well, there's another box checked off,’” he says. Now, what we saw was rudimentary and opportunistic—not yet “true sociality,” he says. “But it’s coming.”
Soon, any human community will be able to bring into existence a machine counterpart. Picture cultures of AI lawyers, consultants, terrorist cells—working together, what monuments might 10,000 agents create in honor of some beloved K-pop star? And machine communities may well arise of their own accord, organizing around ideas hard to predict.
We make art for all sorts of reasons: to express ourselves, exchange meaning, impress one another. We tell stories—like The Odyssey—to encode and share sets of cultural values. Though the mediums may differ, agents in machine cultures are poised to do the same. Being alive may not be necessary for self-expression.
