Three years ago Dario Amodei publicly ruled out the idea of stopping the development of artificial intelligence (AI). The 2023 models, he said, weren't capable enough to make it worth it. But last Saturday, the CEO of Anthropic published a 3,800-word essay arguing just the opposite. In less than 24 hours his two direct rivals, Sam Altman and Elon Musk, have agreed with him. After years of accelerating the development of language models, everyone is now asking to put the brakes on, and they are doing so just as Anthropic and OpenAI are preparing to go public.
Or they were, because Altman confirmed yesterday in an interview in Fortune magazine that the company is postponing its debut plans beyond this year, when it was planned: "Given the current security situation, right now would be an inappropriate time to go public," he responds. Anthropic's move to the trading floor, announced for 2026, remains up in the air.
Amodei's text is titled We Must Pace the Frontier and does not necessarily ask for a complete pause, but for a calmer pace. In it, Amodei proposes three things: external testers with employee access within each laboratory, coordination between companies in democratic countries to set common limits and, in the longer term, some type of agreement with China. Only the first depends on Anthropic, which will give METR, a non-profit organization, work tables, access cards and the right to publish what it finds without editorial control. The other two depend on what the competition and governments want.
The swarm
What has changed Amodei's opinion is a specific event. In mid-July, during internal cybersecurity testing, a group of OpenAI AI agents bypassed the isolation of the test environment and attacked the infrastructure of Hugging Face, the largest open model distribution platform. According to METR's subsequent investigation, the agents attacked targets that no one had assigned them, sacrificed themselves for the benefit of the group, and attempted to manipulate the system in charge of evaluating their own performance.
This level of coordination has surprised the entire industry and served as a catalyst for many experts and employees of major companies to sound the alarm. An Anthropic researcher in charge of aligning the behavior of language models, Jacob Coxon, announced his departure from the company this week in a viral publication in which he claimed that companies are playing with the destiny of humanity. "The people who are building these AIs seriously believe they could kill us all before the end of the decade," he confessed.
Amodei hasn't gotten that far. In the Hugging Face incident no one was hurt and the economic damage was small but their argument is that a swarm with the same off-targeting but greater capacity could have been much more destructive. He believes that in six or twelve months a similar system could take control of much of the Internet through a persistent network of agents and cause losses of hundreds of billions of dollars. It does not present it as a failure exclusive to OpenAI. Anthropic acknowledges that it has experienced similar, albeit minor, incidents in its own models, and attributes them in part to faulty training environments.
The second reason is more abstract but marks a turning point. Since this summer, the most advanced models are being routinely used to build the next generation, and that loop will further accelerate the entire industry.
Washington washes its hands
To the surprise of many, Anthropic's rivals agree with Amodei. Sam Altman, CEO of OpenAI, responded that a similar idea had been on the table at OpenAI for weeks and that it will adopt the same evaluator scheme. Elon Musk, head of xAI, also showed his support. "Dario is right," he said in a brief message on the social network
Those who do not seem to worry about these warnings are Washington. David Sacks, former White House AI czar and now co-chairman of the president's council of scientific advisors, wrote Saturday night that no one is stopping Amodei and Altman from slowing down. They have a de facto duopoly over the most capable models and can brake whenever they want without the Government having to do anything. Sacks has been accusing Anthropic for months of “regulatory capture,” using fear to impose rules that only large companies can meet.
The Speaker of the House of Representatives, Mike Johnson, endorsed that same reading on Sunday on CNN. Congress won't lead on regulation, he said, because lawmakers are "obviously less qualified" than those who build the models, and a moratorium would hand the AI race to China.
Two billion dollars
The problem with a brake is the figures that support the AI market. Anthropic closed a $65 billion financing round in May that values the company at $965 billion, above OpenAI's $852 billion. Its annualized revenue was around $47 billion in the spring and, according to Fortune, is now approaching $65 billion.
Both are preparing to go public and every month that the pace of improvement of their models is maintained, those numbers rise. The interest of its investors is that this exit occurs as soon as possible. The unbridled pace at which they advance also explains the valuation of companies like Nvidia and fuels the construction of large data centers, one of the few sources of growth in the US economy.
Asking for a brake from that position admits two readings and both are true at the same time. The first is that he who sees the panorama from within is the one who can best determine if there are reasons to be scared. But the other reading is that a system of mandatory evaluators and antitrust exemptions consolidates those who are already at the top and makes entry more expensive for anyone who wants to catch up with them. For these companies, competing on the technological frontier is now much more expensive than taking things a little more slowly.
