Anthropic's AI Risks and US Policy Shift

The AI company Anthropic has been making major headlines recently. Its trillion-dollar IPO plan and its blood feud with secretary of defense Pete Hegseth have attracted much attention, but two other events may be even more consequential.

In early June, the company posted an article describing early signs of recursive self-improvement (RSI), a process in which an AI system devises ways to increase its own intelligence, leading to a greater ability to improve itself, and so on.

Obviously, uncontrolled RSI could produce a runaway feedback loop that leads to an irreversible loss of human control. Anthropic suggested the world should “slow or temporarily pause frontier AI development”. Then on 12 June, the White House issued an export control directive banning access to Anthropic’s new frontier models, Fable 5 and Mythos 5, for all foreign nationals – including many of its own key researchers. Anthropic responded by shutting the models down altogether.

These two June events are closely related. A few months ago, Anthropic’s Claude Code became good enough that its leading researchers no longer write any code at all; they just describe ideas and experiments to Claude and it does all the work.

This sped up the cycle of improvement – including the improvement of Claude Code itself – to the point where the latest iteration, called Mythos 5, showed the ability to conduct end-to-end cyber-attacks with no human assistance. If such systems were released without cast-iron guardrails, almost anyone in the world could attack any country’s critical infrastructure at will.

These developments are only to be expected. They are symptoms of the inexorable increase in AI risk arising from the inexorable increase in AI capabilities. Yet, with the honorable exception of the UK’s AI Safety Summit in 2023, the world has largely been ignoring the risks.

The CEOs are telling us: “We’re on track to create superhuman intelligence, which has a good chance of causing human extinction.” (By “good chance” here, they mean a chance similar to the one in six chance of dying while playing Russian roulette with a loaded revolver; in this game, however, the revolver is pointed at all of our heads.) Yet governments reply: “That’s wonderful! Can we offer you a subsidy? Fast-track your permits?”

But finally, with the prospect of weapons of mass cyberdestruction in the hands of billions, the White House has reversed its deregulatory stance and suffered a rare attack of common sense.

They sputter: “Why did no one warn us about these AI systems?” Their response has been spasmodic, with an on-again, off-again executive order and now a ban on a system that had already been deployed, but the direction of travel is clear.

Unrestrained development of unsafe systems leads to intolerable risks. Governments can respond now, before the risks materialize, or they can wait and clean up the mess (if they still exist, that is). One leading AI CEO told me he didn’t expect serious regulation to happen until there was a “Chornobyl-scale disaster”. If that happens, of course, the AI companies can expect to be shut down immediately and perhaps permanently.

The recent changes in White House policy suggest we might not need a Chornobyl to spur real regulation, but perhaps only a Three Mile Island. The kind of regulation we need is not new: a licensing regime that requires a minimum safety standard before a system can be built and released. This is how we handle nuclear power, airplanes, buildings, elevators, hairdressers and sandwich makers. Is it too much to ask of trillion-dollar AI corporations, who claim to be building the most dangerous technology in history?