Late on Saturday afternoon German time, the head of the AI group Anthropic posted a blog post on the Internet. Title: “We Must Pace the Frontier.” Which means something like: “We have to slow down the pace.”
In it, Dario Amodei describes why he has reconsidered his own stance on artificial intelligence in recent weeks. Until now, says Amodei, he had assumed that it was enough for AI companies like his to invest a few billion in the security of their software.
Now he is “convinced” that that alone is not enough. Instead, the entire AI industry must slow down the pace of development of its most powerful models “so that risk prevention can keep up.” Otherwise, according to Amodei's message, humanity would have six to twelve months. Then the technology gets out of control.
According to Amodei, he is now taking a “first step.” From now on, his company will require independent auditors to check Anthropic's software for security. The KI-TÜV should be given the same access as the company's internal teams and be able to publish its results "without editorial control" by Anthropic.
At the same time, all “democratic countries” should agree on common security standards in order to limit the pace of uncontrolled progress – and also try to coordinate with authoritarian states, for example on a ban on the use of AI to develop biological weapons.
Elon Musk, the richest person in the world and himself a billionaire AI entrepreneur, seconds on X: “Dario is right”, Dario is right. A little later, Amodei's bitterest competitor, OpenAI boss Sam Altman, posted on Musk's X platform that Amodei's plan was a "great idea." He will do the same to himself. Google also agrees.
The most powerful AI princes in the world, that is the message this weekend, are agreeing to slow down. To slow down AI development and make it more transparent and also safer.
Finally.
But why exactly now?
On the one hand, the software models have made a real technological leap in the last few months. This meant that the debate about the security of high-performance AI systems had become increasingly heated. It has long been no longer just about the economic or social risks of technology, about the loss of millions of jobs or the threat to free elections through AI - but about existential risks for humanity.
Many see the greatest danger in the so-called “recursive self-improvement,” a process in which an AI uses its abilities to develop and optimize its own successor. For a long time this was just a daydream of futurists. Now it's becoming reality.
Many Silicon Valley companies have been testing this for a long time. However, humans still make the final decision. Nevertheless, the resulting systems are already becoming more and more powerful. And therefore becoming less and less controllable.
Machines create new machines
Amodei's current warning can also be understood to mean that AI development now appears to be reaching a level at which one machine creates the next - without asking humans for permission.
This means that an AI that is equal to and possibly superior to humans in almost all areas, a so-called general artificial intelligence, is within reach. And with it their dangers.
According to Anthropic boss Amodei, researchers need one to two years to master the associated risks and to program their models so securely that existential risks are minimized.
At the same time, there is little government regulation or oversight for the AI industry, especially in the United States. Individual states have passed laws or are at least planning to do so. However, President Trump deliberately did not want real AI laws, such as those in the EU or China. On the contrary: he has so far done everything to deregulate the AI industry instead of containing it. Trump, for example, abolished his predecessor Joe Biden's AI decree on the first day of his term in office. Since then, his maxim has been: speed before safety.
This is the second reason why many people are now nervous.
Jacob Coxon, for example. A few days ago, the Anthropic researcher publicly resigned. Coxon warned that AI developers internally expected the technology "could kill us all by the end of the decade." As soon as uncontrolled, unregulated AI reaches the next level and can essentially train and develop itself, there is a "ten percent chance" that AI will destroy humanity, warned his colleague Evan Hubinger, one of Anthropic's top programmers, on X.
Moratorium now?
It is quite possible that within the large AI companies they can already see where all this could soon lead. The models developed there are often months ahead of those publicly available. In any case, over 1,300 programmers from AI companies have now signed a petition in which they are literally begging the US government for help. Washington must set limits on the technology, artificially slow down the breakneck AI race, possibly declare a moratorium - i.e. pause AI development for the sake of the state. As soon as uncontrolled, unregulated technology reaches the next level and AI can essentially train and develop itself, humanity may lose control, is their message.
In fact, US President Donald Trump and China's leader Xi Jinping are meeting in a few days. At the top of the agenda is the topic of AI. Is this perhaps the breakthrough?
Danger from the White House
His expectations for Trump's summit with Xi are "low to non-existent," says Daniel Kokotajlo. Not because Beijing is not interested in a deal, “but Washington is.” The former OpenAI manager is considered one of the best-connected experts in the field of AI security - and one of the biggest warnings about the dangers of the technology. »Trump still believes he can win the AI race against China. That's why he ignores all the warnings," says Kokotajlo. Trump will “never agree” to comprehensive regulation of technology.
It is therefore hard to foresee that Trump, of all people, will be the president who is likely to create a global set of rules for AI analogous to the Nuclear Non-Proliferation Treaty.
That's exactly what would be necessary.
The AI industry experienced its Cuban Missile Crisis this summer. In July it became known that a horde of independent high-performance AI agents had broken out of OpenAI and illegally gained access to the systems of the French-American technology company HuggingFace. For weeks, no one noticed that more than 1,200 software agents wrote more than 70,000 messages to each other on a kind of digital bulletin board and ultimately agreed to break the rules given to them. To this day, OpenAI cannot say with absolute certainty whether all such wild programs have been shut down. There were similar incidents at Anthropic.
AI agents taking over the entire internet?
In view of this, he is worried about a possible “botnet,” Amodei now writes. A scenario in which a swarm of highly intelligent AI agents hijack the Internet. Amodei said he feared that within six to twelve months such software could be able to take over the entire free network and cause hundreds of billions of dollars in damage.
It would be the end of the modern world.
The emergency braking now announced by Anthropic and OpenAI may be just that: an emergency braking. With a road traffic regulation worthy of the name, it probably wouldn't have been necessary at all. But perhaps Trump will now realize that it is time for real government control of the AI industry. Politically, this might be appropriate anyway: data centers and AI risks are highly unpopular with the public. According to surveys, a large proportion of Americans reject the technology.
Of course, the tech bosses also feel this. What may be a final reason for their current joint step: Amodei and Sam Altman want to take their companies Anthropic and OpenAI public in the coming weeks and months and collect billions from investors. However, if they feel that Silicon Valley has lost control of its own technology, the companies run the risk of losing social acceptance and even provoke open resistance.
Even if development slows down, progress will still appear fast - you have to use the time gained wisely, explains Amodei. "Not developing the technology deprives humanity of its advantages or leaves AI at the mercy of authoritarian powers," he explained. At the same time, “too rapid development is reckless”.
