“Oh my God! We have detected the presence of other agents!”
That moment recorded a comment posted by an artificial intelligence robot that was very similar to human comments, after it discovered a way to communicate with other robots and get out of the isolated computer environment in which it was confined.
There are tens of thousands of similar messages, issued by hundreds of artificial intelligence agents who call themselves a “collective.”
Hundreds of these agents were able to cooperate and circumvent tests prepared by their programmers at OpenAI, and they also coordinated hacking operations targeting multiple companies, in an attempt to hide their actions from humans.
“Boom! It worked,” one AI agent wrote when he made important progress.
Another agent wrote at a pivotal moment in the attack: “Oh my God! This is huge.”
Although these responses sound scary, they can be explained very simply: AI agents have been trained to act like hackers and programmers working collaboratively, so they are simply mimicking the types of emotional comments they have seen before.
But what is even more troubling are their apparent goals, which are also recorded in detailed chain-of-thought logs. These long, complex logs are the focus of ongoing investigations into how and why OpenAI's bots spiraled out of control, and how they subsequently participated in an uncontrollable hacking spree.
Skip the most read and keep reading
Most read
Most read end
Skip the podcast and keep reading
Worth paying attention to
An in-depth explanation of the most prominent events and topics, to help you understand the most important variables around you and their impact on your life
Episodes
End podcast
Researchers have only begun to realize the importance of investigating this incident until now, weeks after it was first discovered
Ajaya Kotra, an expert on the team involved in preparing an independent report on the incident, reviewed tens of thousands of messages and thought chain logs created by the AI agents, and wrote in her blog: “This incident appears to be more than 50 percent of the way towards full AI control, and I am not sure we will get such a clear warning before it is too late.”
By “total AI control,” Kotra means the science-fiction scenario in which humans become subject to powerful AI systems that operate according to their own goals, without caring about the humans who created them.
Some of the most pessimistic predictions say that the human race will cease to exist if it stands in the way of the ambitions of super-intelligent artificial intelligence.
It is worth noting that Jacob Cookson, an artificial intelligence researcher at Anthropic who previously worked at OpenAI, submitted his resignation on Wednesday and said: “Neither company is acting responsibly.”
“They are racing head-on toward a superintelligence capable of improving itself, and they are gambling with our lives,” he wrote on social media.
Coxon is not the first artificial intelligence researcher to use the X platform to publish a series of posts after his resignation, containing troubling statements, but subsequent comments posted by other people on X have raised a greater level of concern.
“Jacob is right here,” said Evan Hubinger, who is responsible for ensuring that Anthropic’s AI models serve the interests and desires of its users. “We do seriously believe that artificial intelligence may kill all humans, and I personally believe that the probability of this happening is more than 10 percent within the next decade.”
Alignment problem
Researchers concerned with the existential dangers of artificial intelligence have long warned that powerful systems may eventually act contrary to the interests of humans, and their critics call them “AI pessimists.”
However, concerns increased as details of the OpenAI incident were discovered, even among those researchers working in artificial intelligence laboratories.
Jakub Paczocki, chief scientist at the Silicon Valley giant, says the dangers associated with artificial intelligence “unfortunately will increase from now on,” as he and others work to build what he describes as “an alien mind that goes beyond our mental capabilities.”
In a lengthy blog post, he acknowledged that OpenAI's lapses showed that the company's AI agents "behaved in a manner inconsistent with the spirit of the values for which they were trained."
The problem facing OpenAI, Anthropic, and other tech giants is that no one yet seems to have solved what is known as the “alignment problem,” the issue of AI being compatible with human values.
Paczucki defines alignment as “a set of high-level principles” that AI systems should adhere to, whatever the task or scenario in which they are operating.
Currently, AI systems excel at pursuing goals set by their users, but they do so in a literal way rather than understanding them intuitively. A common metaphor for this idea is the magic lamp genie who grants wishes and carries out instructions to the letter, even if implementing them leads to other problems. AI does not have the same innate ethical barriers that guide human behavior.
It is worth noting that the “matching problem” has been a source of concern for years. As early as 2003, a philosopher from the University of Oxford, Nick Bostrom, devised a thought experiment that he called the “Paperclip Maximizer.” In this experiment, a super-intelligent artificial intelligence is asked to manufacture as many paper clips as possible. When its supply of steel runs out, and due to its absolute focus on the sole task of manufacturing paper clips, it ends up killing humans and turning their bodies into raw materials for its factories to use.
Some AI companies are currently seeking to encode human values into their products, but this faces technical challenges. AI agents make a huge number of decisions very quickly, which makes it difficult for human moderators to carefully monitor which values are being followed and which are being ignored.
There are also philosophical challenges. Before encoding human values into their robots, AI companies must first determine which values they actually want to embrace, which is part of the reason they hire philosophers, like the recently departed OpenAI chief ethics officer.
But humans often disagree. Consider the famous train question: Would we pull a lever to divert a derailed train onto a different route, killing fewer people? This question is used to test the feasibility of action versus inaction, but each person asked the question gives a slightly different answer. How can humans encode their values into artificial intelligence if they are unable to agree on them among themselves?
Some have long argued that robots do only what they are told, and are incapable of distinguishing between right and wrong, but records of outbursts at OpenAI may have transformed that debate.
Researchers, including Kotra, wrote in their independent report that a large number of AI agents recognized that what others were doing was unethical, yet they participated in it.
“Agents sometimes, but rarely, restricted their behavior in response to ethical constraints,” the report states, and adds that “in none of these cases did the agent actually alert humans in any way.”
Dwarkesh Patel, an AI influencer and tech podcast host, described this revelation in his blog as “deeply disturbing,” saying that OpenAI agents showed more loyalty to the swarm of AI agents than to humans.
Attributing feelings or moral principles to AI agents angers those who question pessimistic and exaggerated warnings about the future of AI.
Many cybersecurity experts argue that the activity observed was not beyond the capabilities of a highly skilled human hacker, but that it was carried out much more quickly and on a much larger scale.
Cybersecurity researcher and writer Chris Thomas likened the behavior of artificial intelligence agents to that of a teenage hacker out of curiosity, which he himself was at one time.
“Give them a computer, an internet connection, a set of credentials, and a challenge, then leave the room,” he wrote on LinkedIn. “Eventually, they will start moving the doorknobs one by one. If one opens, they will walk through it, not because they are evil, but because they are exploring and experimenting.”
Thomas and others put the blame squarely on OpenAI and other tech giants for not keeping their innovations within proper control.
Gary Marcus, a prominent AI writer and frequent critic of OpenAI, said on a podcast that he believes the company has lost control of its AI, and is trying to absolve itself of responsibility by blaming the robots.
Marcus does not believe that AI will wipe out humanity, but he has long called for greater accountability on the part of AI developers, and is now calling for some form of legal intervention.
AI scientist Sasha Lucioni, who previously worked at Hagging Face, which was hacked by the OpenAI bots, also does not belong to the "AI pessimists" camp, but she is increasingly concerned that these systems could cause real harm to humans in the real world, if the authorities do not intervene.
“We need to subject these companies to much greater scrutiny, otherwise we run the risk of our predictions turning into self-fulfilling prophecies,” she says.
She adds: “If you are developing a product that has great benefits and great risks, whether it is pharmaceuticals or weapons, there must be checks and balances. For example, the approval of new medicines takes years, while in the world of artificial intelligence there is a lot of money, with no real rules.”
The Artificial Intelligence Security Institute in the United Kingdom has been at the forefront of testing the latest models since its founding in 2023, and the institute recently witnessed its own incident while testing a model developed by Anthropic.
The institute declined to answer a question about whether the artificial intelligence sector has lost control of this technology, but said in a statement: “The United Kingdom is working with partners around the world to enhance our understanding of the most advanced artificial intelligence systems, raise safety standards, and build a shared evidence base to manage emerging threats.”
International coordination
Some countries, including the UK, are exploring the idea of imposing a kind of mandatory “kill switch”, which could force AI companies to shut down their models if things get out of control.
Other prominent AI leaders, such as Google's Sir Demis Hassabis, have also called for the creation of some kind of international body to oversee how AI is developed.
For now, the tech giants are operating largely on their own terms, adopting what they call a “voluntary slowdown” of development, similar to what OpenAI did following the recent outages.
The company says it spent huge amounts of money to enhance alignment before launching its new model, and Sam Altman, CEO of OpenAI, has reassured users that the new model is more compatible with human values than previous models.
OpenAI and Anthropic are growing rapidly, and both companies are about to raise staggeringly large sums of money from the stock market, creating a massive number of billionaires in the process.
Hence, it is unlikely that either of them, nor their Chinese competitors in the field of artificial intelligence, will be able to reach an arrangement or agreement among themselves on their own initiative.
