There are few sentences that silence a room as reliably as this one: “The risk of AI wiping out humanity is over ten percent within the next decade.”
The phrase comes not from an activist or a science fiction author, but from Evan Hubinger, the head of alignment research at Anthropic. This is the company that sees itself as the most security-conscious laboratory in the AI industry. At the beginning of September 2026, Hubinger left the company and made his assessment public. Samuel Marks, Anthropic's head of scalable oversight, acknowledged: "The higher an employee's position, the greater the concern." Jacob Coxon, who worked at Open AI and Anthropic, was even clearer: Both companies “are racing toward self-improving superintelligence and playing with our lives.”
These were not isolated voices. A few days later, Anthropic boss Dario Amodei published an essay entitled “We Must Pace the Frontier.” In it, he described two developments as imminent: recursive self-improvement, which means that AI systems design their own, more sophisticated successors. And autonomous cyber attacks without human intervention. Sam Altman, head of Open AI, joined in, promising independent investigators with “employee-like access” to his labs. Elon Musk wrote on X: “Dario is right.” The Financial Times, a paper that is rarely prone to alarmism, called in an editorial for a pause in the development of frontier models and criminal liability for the heads of AI companies. More than 1,300 employees from AI companies signed an open letter.
At the same time, polls from the University of Maryland show that in the US, 82 percent of Democrats and 78 percent of Republicans support government security testing for AI. 85 or 79 percent want a new federal authority for AI regulation. So this is not an issue on which the public would be divided.
And yet there is neither a law nor a binding international treaty nor an authority with the right to intervene. The US government's only action - President Trump's Executive Order of June 2026 - provides for a voluntary, 30-day preliminary review of new models and contains an express clause prohibiting any mandatory licensing. President Biden's broader 2023 Executive Order was revoked by Trump on his first day in office.
How is that possible? How can a threat deemed existential by the brightest minds in the industry and felt urgent by the population elicit such a weak response?
The explanation does not lie in a single failure, but in the interaction of two structural problems: one of game theory and one of power politics. Both reinforce each other - and together they make any rapid, effective regulation almost impossible.
I. The Prisoner's Dilemma: Why Reason Is Not Enough
To understand why even clear warnings have no effect, a concept from game theory helps: the prisoner's dilemma. In its basic form, it describes two suspects being interrogated separately. Everyone can remain silent or burden the other. If both remain silent, they will get off with a light sentence. If one incriminates the other while the other remains silent, the traitor goes free - the silent one receives the maximum punishment. Since no one knows what the other is doing, it is rational for each to burden the other. The result: Both incriminate each other and receive a high punishment - even though silence on both sides would have been better for both. Individual rationality leads to the collective worst outcome.
The Oxford computer scientist Michael Wooldridge gave the AI race exactly this game theory name in May 2026 - albeit with more than two players. Every company – and every country – would be better off if everyone slowed down. But no single actor can afford to unilaterally slow down because they have to assume that the others will continue. “We have a small number of very, very rich companies pushing AI and at the same time saying they are afraid that something could go terribly wrong,” Wooldridge said. "Then why are they pushing it? Because they believe that if we stop, someone else will."
In the classic prisoner's dilemma, mutual development - mutual "defection" - is the Nash equilibrium: the dominant strategy for each player, regardless of what the others do, even though mutual restraint would produce a better outcome for everyone. This formalizes a striking empirical observation: prominent calls for moratoriums – from Turing Prize winners Bengio and Hinton, from the growing Pause AI movement, from the Effective Altruism community and now from Anthropic itself – have so far had no observable impact on the speed of development. Security research accounts for less than three percent of all AI publications. And even Amodei, who publicly criticized his competitors' spending, pledged $100 billion in investments to Amazon four months later.
The parallel with arms control is obvious – and it is disheartening. During the Cold War, superpowers managed to limit the number of nuclear warheads through treaties such as SALT and START. This worked because nuclear weapons have a physical property that enables verification: you can count missile silos, detect test explosions with seismographs, monitor launch pads with satellites. Artificial intelligence does not have this property.
“Training runs are far easier to hide than missile silos,” wrote Jack Clark, co-founder of Anthropic, when his company became the first major AI laboratory to call for a global development pause in June 2026 – on the condition that it be observed simultaneously and verifiably by all leading laboratories. The condition itself illustrates the trap: even the company with the strongest security credentials recognizes that unilateral restraint within the existing competitive structure is irrational.
In his September essay, Amodei proposed a way out that builds on the example of the SALT negotiations: a four-tier coordination system from unilateral commitments by individual laboratories to bilateral agreements between the US and China to a global framework. Its core instrument is a “checkpoint system”: from a certain skill level, an AI model would have to have corresponding safety certificates, similar to how new drugs require approval before being sold.
Martin Wolf, economic commentator for the Financial Times, agreed - and at the same time pointed out that the attempt "will most likely fail." Niall Ferguson, a historian at the Hoover Institution, had previously argued that a bilateral AI disarmament treaty between the US and China was the necessary prerequisite to curb the race between companies. But Ferguson also sees verification as the decisive obstacle.
The irony is brutal: Anthropic called for the break - and at the same time had engineers stationed in the NSA who carried out offensive cyber operations with the high-performance Mythos model. The reasoning: “If Mythos is not used to build attack agents, adversaries will find a way to do it themselves.” The sentence is the perfect summary of the prisoner's dilemma: the same company advocates for common restraint and at the same time deserts on the offensive dimension.
II. The battle of the factions: Why Trump doesn't act
Even if the game theory obstacles could be overcome, a second blockage would stand in the way: the internal turmoil of the Trump administration. Behind the façade of a unified deregulation agenda, at least five distinct factions are vying for influence - with shifting alliances and conflicting goals.
The growth faction is the most powerful. Its architect was David Sacks, who as “AI and crypto czar” designed the government’s AI Action Plan before formally stepping down in March 2026 to become PCAST co-chair. Behind Sacks are venture capitalists like Marc Andreessen and the AI industry lobby, which now employs 25 percent of all 13,000 federal lobbyists in Washington - a quarter of the country's entire lobbying apparatus. Sacks' policy is radically deregulatory: In September 2026, from Trump's Doonbeg golf course, he shouted at companies to "stop acting like they need anyone's permission." He interpreted security warnings from the industry as “blackmail of the public and the political system” – an attempt to buy a regulatory system that keeps competitors out. This faction blocked the signing of the original executive order in May 2026, forcing the dilution to a voluntary thirty-day period and enforcing the anti-licensing clause.
The security pragmatists around Treasury Secretary Scott Bessent, Chief of Staff Susie Wiles and Secretary of War Pete Hegseth are not acting out of technology policy conviction, but out of political risk management: If a spectacular AI-supported cyber attack takes place under a laissez-faire policy, “it will look as if the government made it possible.” This faction was activated by the myth shock in June 2026 - when calls from Microsoft CEO and JPMorgan boss Jamie Dimon reached the White House. Wiles was the key institutional player that kept the executive order alive after the initial cancellation.
The regulatory faction – represented by Kevin Hassett, director of the National Economic Council – went the furthest: Hassett proposed mandatory prior authorization along the lines of the FDA. No senior U.S. official has ever been as close to meeting the demands of academic security researchers as Yoshua Bengio. But Hassett was the first to lose: his proposal was replaced by the ninety-day deadline, then dropped completely, and was finally replaced by a thirty-day version with an anti-licensing clause - a massive dilution.
The coalition for mandatory control has expanded dramatically since the summer of 2026. Its core was the Humans First coalition: Steve Bannon, MAGA leaders, conservative organizations and 38 evangelical pastors, linked to the Future of Life Institute as a bridge between the Effective Altruism movement and the populist right. Their open letter called for “mandatory testing, evaluation, verification and government approval” – even stronger than the original ninety-day draft.
In September 2026, this coalition grew into the “Pro-Human Assembly,” organized by the Future of Life Institute in Washington: Bernie Sanders and Steve Bannon spoke at the same event (though not on the same stage), along with Republican Chip Roy, Democrat Greg Casar, Hollywood actors, union leaders, and clergy. Sanders introduced a bill that would ban the development of “artificial superintelligence.” That the democratic socialist and the right-wing populist are in a coalition is a remarkable bipartisan alliance, almost unprecedented in recent American politics. They are united by opposition to concentrated technological power, with fundamentally different ideological foundations.
The state functionalist faction around Alex Karp and Palantir sees AI as the successor to nuclear deterrence - as a strategic instrument that must remain in state hands and be under military control. At the same time, Karp signed the open letter against restrictions on open source models, highlighting the conflicting loyalties within the administration.
Perhaps the most surprising new voice comes from Vice President J.D. Vance. He called the industry's demands for regulation "a kind of Trojan horse." Companies used security rhetoric to cement their market position. This is not a classic anti-regulation position, but rather a populist distrust of the regulators themselves, which further fragments the political space.
The fragmentation explains why there is no coherent response coming from the White House. Trump himself reacts to the situation: In Doonbeg he brushed off security concerns - "things that won't happen" - and emphasized competition with China: "Whoever wins AI wins everything." But three months earlier, he had almost signed a pre-clearance executive order before Sacks talked him out of it at the last minute. And three days after signing the watered-down version, he announced that the government would acquire equity stakes in AI companies - a proposal that Sacks condemned as a "CCP-style social credit system" but failed to prevent. An Economist report called the result “murky and untenable”: The Office of the National Cyber Director is losing senior staff, the Commerce Department has limited in-house technical expertise, and government AI expertise has continued to shrink over the past 18 months.
III. The double blockade
Both structural problems reinforce each other. The prisoner's dilemma at the international level gives the growth faction its strongest argument: any slowdown in the USA is a gift to China. “Whoever wins AI wins,” Trump said – and Mike Johnson warned that a moratorium would let China “overtake us.” The prisoner's dilemma is not just an abstract game theory construct, but a political weapon in the hands of those who reject any regulation. Conversely, domestic political fragmentation prevents precisely the institutional capacity that would be necessary to break through the prisoner's dilemma at the international level: binding agreements, verifiable standards and authorities capable of acting.
What is striking is who is missing from this debate: Europe as an independent actor. This is because the continent does not play a role in the development of frontier models. It is also weak in terms of safety structures: there is no European equivalent to the British AI Safety Institute and no assessment capacity for frontier models. The EU AI Act, often celebrated as a milestone, regulates the use of AI systems – not their development. Laboratories in the USA and, in the future, China decide whether a new model can be trained.
