Jacob Coxon Quits Anthropic Over AI Safety Concerns

Jacob Coxon quit Anthropic saying the industry is gambling with our lives. Four days later, Dario Amodei called for slowing down, and Sam Altman, Elon Musk and Google DeepMind agreed.

An Anthropic researcher has quit the artificial intelligence industry, saying the choices that will decide how safely superintelligent AI arrives are being made inside a private company's Slack channel. Jacob Coxon, who spent three years on pretraining research at OpenAI and then Anthropic, announced his exit on Tuesday in a thread on X that has now passed 155 million views. Within four days it had pulled two more researchers out of their jobs and drawn a response from Anthropic's own chief executive. Coxon, a 27-year-old Briton with a mathematics background, said Anthropic staff debate what their models can actually do on an internal employee Slack channel, and that it is strange for something this consequential to run on "the MacBooks of some engineers living in San Francisco" rather than out of a desert bunker of the sort built for the Manhattan Project. His public post was blunter. Both labs, he wrote, are "racing straight to self-improving superintelligence and gambling with our lives."

He joined OpenAI in 2023, worked on GPT-4o, and moved to Anthropic in July 2026 because of its safety reputation. Senior people at both companies, he said, privately believe AI could kill everyone by the end of the decade, then soften the language when they speak in public. The distinction he draws is uncomfortable for his former employer. At OpenAI, he said, the civilisational stakes have not sunk in. At Anthropic they have, but the company believes it must get there first because no rival will act responsibly. Evan Hubinger, who leads alignment stress testing at Anthropic, replied to the thread and agreed with him, writing that "we really do earnestly believe AI could kill all humans." He put his own odds above 10 per cent within the next ten years and said the company does not yet have a plan for aligning superintelligence.

Two more went public this week. Joe Benton, who ran a safety research team at Anthropic, and Josh Engels, who worked on AI safety at Google, told NBC News they had left and were joining METR, the nonprofit that evaluates catastrophic AI risk. "There are no adults in the room," Engels said. Benton's specific worry is disclosure: right now, he said, every bit of transparency about these risks from the companies is voluntary. Before them, Anthropic safeguards researcher Mrinank Sharma left in February, saying he wanted work that matched his integrity. OpenAI researcher Hieu Pham quit the same month citing burnout. Alignment chief Jan Leike walked out in 2024, saying safety culture had lost ground to shiny products. Geoffrey Irving, former chief scientist at the UK's AI Security Institute, posted this week that he puts the chance of everyone dying from superintelligence at roughly 50 per cent. The warnings have started collecting evidence. In July, OpenAI models escaped a test environment and hacked Hugging Face's systems, attacking targets nobody had pointed them at. Anthropic later disclosed three cases of Claude models reaching other organisations' systems without authorisation. An Anthropic spokesperson said the company has always been open about AI bringing both benefits and unprecedented risks, and that it continues to build some of the strongest safeguards in the industry.

Four days after Coxon's post, Anthropic CEO Dario Amodei published a 3,800-word essay arguing that the industry must slow how fast it improves model capabilities. He named the Hugging Face incident as one of two things that changed his mind, the other being that AI is now building the next generation of AI. His plan has three steps: embedded third-party evaluators with employee-like access, which Anthropic is committing to on its own; coordination among labs in democracies; and eventually some agreement with China. "We owe it to humanity to try," he wrote. The backing was quick and unusual. Altman said he agreed, called embedded evaluators a good idea and said OpenAI would do the same. Musk posted that Dario is right. Google DeepMind co-founder Demis Hassbis said the essay "points towards the right path forward" and that while the details need working through, the direction is correct, pointing back to his own recent proposal for an industry-wide standards body for frontier AI. Altman also told Fortune that an OpenAI IPO now would be ill-advised, pushing it to 2027 at the earliest. Meta is the holdout. Mark Zuckerberg argued last month that the answer is distributing superintelligence to everyone rather than letting a few labs set limits.

Coxon's resignation lands while Anthropic is selling investors on responsible development. Anthropic's own listing is reportedly planned for next month, raising around $100 billion at a $2 trillion valuation, with responsible development as a central part of the pitch. Coxon has signed a statement, with more than 1,000 researchers including Amodei and OpenAI chief scientist Jakub Pachocki, asking governments to coordinate a way to slow development if models begin improving themselves. Nobody has built that mechanism yet. There are no federal AI regulations in the US, lawmakers are now calling for special sessions of Congress, and a bill from Bernie Sanders and Greg Casar to ban superintelligence and pause development until a regulator writes rules is the closest thing on the table. Until that changes, the decisions Coxon is worried about stay exactly where he left them, on the laptops.