An artificial intelligence (AI) researcher who has left Anthropic told the BBC that people working on the technology are "genuinely frightened" by the speed of its progress and its possible impact on humanity.
Jacob Coxon was interviewed by the BBC's Laura Kuenssberg on Saturday (September 12) about his widely publicized resignation post, in which he raised concerns about runaway artificial intelligence.
"I believe that if we don't slow down the current pace of development, there is a good chance that we will all die in the near future," he said.
His resignation comes amid growing concerns about safety in the industry, including warnings from his former boss, Anthropic President Dario Amodei. In an article published on Saturday, Amodei advocated for slowing down the development of artificial intelligence.
However, some in the industry believe the dangers are exaggerated, possibly to build momentum ahead of the potential IPOs of two of the largest artificial intelligence companies or to prompt authorities to introduce regulatory measures that could slow down rivals.
Coxon, 27, from the UK, focuses his research on training artificial intelligence models. He said that artificial intelligence leaders including Elon Musk and OpenAI’s Sam Altman have expressed similar concerns to Amodei.
"They've all spoken out about the need to be cautious about the potential for AI to take over the world," Coxon said. "They've all talked about it. An AI taking over the world would mean the extinction of the human race."
The hardest question to answer, according to Coxon, is what form this situation will take.
"Any scenario you can describe in detail ends up sounding like science fiction," Coxon said. But he pointed out that many of the technological advances that have been realized would have been considered fiction in the past.
One potential scenario, Coxon said, is that artificial intelligence agents invade medical laboratories to "create deadly viruses autonomously." Another example he cited was artificial intelligence invading "critical infrastructure that the world relies on to function."
The researcher pointed to a recent OpenAI report detailing how its technology launched a series of hacking attacks on an online platform called Hugging Face.
"They did it autonomously without human encouragement. They chose to launch this hack on their own," Coxon said. "It's a combination of things getting faster and things getting scarier."
Anthropic President Amodei proposed a "three-point plan"
Reporting by BBC reporter Paulin Kola and BBC North America technology reporter Lily Jamali from San Francisco
In an article published on Saturday titled "We Must Pace the Frontier," Dario Amodei proposed a three-point plan, including independent monitoring during the development of artificial intelligence models, promoting industry-wide regulation, and establishing a global regulatory mechanism.
The Anthropic CEO later discussed the impact a slower pace of development will have on the industry and competition with leading global developers, particularly China.
"I believe that if slowing down gives us an extra year or two before the model reaches a critical capability level, and we use that time to move forward with alignment, we can significantly reduce the risk of serious problems," Amodei said.
He said this must be done in a coordinated manner "without sacrificing commercial advantages or U.S. leadership in artificial intelligence." He said any slowdown must be limited to avoid allowing China to catch up.
He urged the U.S. government to take measures to ensure that U.S. companies will not sell artificial intelligence chips to China or share related technologies with authoritarian countries.
As the proposal spread widely, even competitors expressed support for the creation of a third-party watchdog to assess the safety of the models as they were developed.
"I agree with Dalio that we need to control the pace of cutting-edge development," OpenAI CEO Altman wrote on X, calling independent evaluators "a very good idea."
Musk said the Anthropic chief was "right."
The warnings have prompted calls for action, but U.S. President Donald Trump has so far dismissed such concerns. He said on Thursday (9th) that he was worried that "if we cannot win in the field of artificial intelligence, we will be in a very disadvantageous situation."
One of the risks outlined in Amodei's remarks is that a swarm of robot programs operating like supercomputers could take over the Internet.
Coxon said that scenario could become a reality within six months to a year.
In response to Coxon's departure, an Anthropic spokesperson told the BBC: "We have always been honest that artificial intelligence will bring both huge benefits and unprecedented risks. To address these risks, we continue to build models with one of the strongest safeguards in the industry."
The spokesperson added that the company has been a pioneer in studying how artificial intelligence models operate and was the first to publish a framework to mitigate related development risks. It will "actively" test its models and publish results to assist outside scrutiny and prevent "artificial intelligence inaccuracies" incidents.
"Because of this, we believe the world would benefit if the industry collaborated in a legal and verifiable manner to control the pace of release of powerful models."
Anthropic and OpenAI are reportedly preparing for what could be a record-breaking initial public offering (IPO). Industry insiders believe that remarks about the dangers and power of artificial intelligence may be aimed at creating market momentum. Other critics say Anthropic has been trying to push for regulatory action to hinder competition and create a duopoly with OpenAI.
Coxon’s views were also questioned by Clement Delangue, CEO of artificial intelligence platform Hugging Face.
"Sorry," he wrote on
However, following Amodei's article on Saturday, Delangue offered to help participate in potential solutions proposed by Anthropic leaders.
Multiple participants told the BBC that Jensen Huang, CEO of NVIDIA, also mentioned Coxon's remarks in front of the audience at a conference hosted by investment bank Goldman Sachs last week. They said Huang believed the claims were untrue.
Jen-Hsun Huang has previously said that the idea that artificial intelligence "will end humanity" is "completely ridiculous."
While he may have a business stake in the AI boom because Nvidia makes the chips that power AI systems, his comments reflect a growing backlash in Silicon Valley over warnings of an existential threat to current and former AI practitioners.
Coordinate with China?
Any slowdown in research and development will require coordination among technology giants, while also taking into account technological advances from China. Coxon says that may be the problem.
"People who work in these companies are very serious when they ask for regulation because they find themselves locked in a race. And they're afraid of the outcome of that race."
He said that these practitioners are "worrying about the fate of mankind in the next two years" and are "planning their own lives."
Still, Coxon remains somewhat optimistic about the future of artificial intelligence. He told the BBC that those studying the technology "really want to see the positive side" of it, such as using artificial intelligence to "cure diseases and improve everyone's lives".
His remarks about artificial intelligence were shared millions of times on social media, and Anthropic scientist Evan Hubinger responded on X agreeing with Coxon.
Hubinger said: "We do sincerely believe that artificial intelligence may kill all humans! I personally think the chance of this happening within the next ten years is more than 10%."
Leading figures in the field of artificial intelligence have been sounding the alarm for years about the security threats posed by the technology. As early as 2023, the directors of OpenAI, Google DeepMind and Anthropic had expressed similar views.
The warnings have grown louder in recent weeks as evidence mounts that companies may struggle to control the technology.
Some in Silicon Valley shrug off tough warning
BBC North America technology reporter Lily Jamali reported from San Francisco
Every September, well-known corporate executives from all over Silicon Valley gather at the Palace Hotel in San Francisco to showcase their charm to investors at a conference hosted by the investment bank Goldman Sachs.
In addition to discussing growth and potential returns over the past week, tech giants also had to respond to the sudden resignation of Anthropic researcher Coxon.
Coxon is far from the first AI insider to issue a public warning. Both Anthropic and OpenAI have seen a string of high-profile departures in recent years, apparently related to security concerns; some current Anthropic employees even echoed Coxon's post.
Although Coxon made it clear in the post that his warning was "not marketing," some Silicon Valley executives and investors are skeptical of the recent wave of warnings from industry insiders.
Anthropic was valued at US$965 billion (6.47 trillion yuan; NT$30.53 trillion) in a funding round earlier this year.
The BBC has sought response from Anthropic for the relevant comments.
Some critics have even accused Anthropic of spreading fear in the hope of pushing for regulatory measures that would exclude competitors and create a duopoly in the field with OpenAI.
Brad Gerstner, head of the investment company Altimeter Capital, posted a photo of Huang at the meeting on social platforms and accused Coxon of "ridiculous exaggeration."
Under the glow of chandeliers beneath the Palace Hotel's stained-glass domed atrium, some investors told the BBC that comments by Coxon and others could accelerate the pace of government tightening of regulations.
