Former AI researchers warn against Open AI and Anthropic

Jacob Coxon previously worked on developing artificial intelligence at rival giants Open AI and Anthropic, before recently resigning from the latter. On Tuesday, he went out on platform X to warn his former employers.

"None of the companies are acting responsibly. They are racing full speed towards self-improving superintelligence and they are playing with our lives," writes Coxon.

Self-improving superintelligence is the point at which an artificial intelligence has become superior to the human mind, and has also learned to rewrite its own programming code so that it evolves and becomes increasingly competent. It's a hypothetical scenario that many AI critics have warned against over the years.

"Don't underestimate the power of this technology. There will soon be super-intelligent systems that can hack anything, revolutionize any field overnight, and acquire real power and resources," writes Jacob Coxon in a post that, at the time of writing, has been viewed over 140 million times.

In an interview with the tech magazine Wired, Coxon develops his reasoning and says that he believes that artificial intelligence can wipe out humanity or create a devastating disaster by, for example, developing new, powerful biological weapons or carrying out massive cyber attacks against critical infrastructure.

He also receives support from a somewhat unexpected quarter. Evan Hubinger, who leads a team at Anthropic working on creating safe AI, agrees with his former colleague's warning.

"Jacob is right - we truly believe that AI could kill all humans! Personally, I think the risk of that is over 10 percent in the next decade," writes Hubinger on X.

Microsoft founder Bill Gates recently warned that the threat posed by AI is far greater than the tech industry is letting on, and UN human rights chief Volker Türk recently came out saying that artificial intelligence could become an "existential threat" to humanity.

The debate about whether AI poses such a threat took off in earnest after Swedish philosopher Nick Bostrom's book Superintelligens: The Age of Thinking Machines, which was released in 2014. The warnings of a disaster scenario have since become increasingly vocal after the great technological advances in AI development in recent years.

Fredrik Heintz, who is a professor of computer science at Linköping University, himself researches artificial intelligence and has followed the debate for a long time. He says the kind of warnings Jacob Coxon and Evan Hubinger are now raising are nothing new.

- But the threat to humanity is not the risk I am most worried about when it comes to artificial intelligence. At the same time, you can't say it's impossible, he tells DN.

He believes that there are significantly bigger and more urgent problems to tackle when it comes to artificial intelligence. Areas where even today's models can be used to create great damage.

- I think that cyber security, for example, is a much bigger risk. Malicious actors can use AI models to get into IT systems that they should not have access to, which could have very large negative consequences, says Fredrik Heintz.

He himself is unsure whether we will even achieve the type of superintelligence that some scientists are now warning about, and at the same time questions the image that it would mean an existential risk to humanity.

- People always seem to expect that a superintelligent AI would become evil, but to me it is not obvious that even if we get a superintelligence, it would mean that it would deliberately try to harm us, says Fredrik Heintz.