The resignation of a former researcher at Anthropic sparks widespread debate about the dangers of artificial intelligence

Jacob Cookson, a former researcher at Anthropic, opened the door wide to a wave of criticism and controversy.

Coxon resigned last week, announcing this through a series of widely circulated blog posts on the “X” platform, in which he accused artificial intelligence companies of not acting responsibly, despite all of their belief that artificial intelligence may lead to the extinction of humanity.

Some media reports had previously published some predictions about what was happening throughout Silicon Valley, but what Coxon published sparked a massive wave of pressure on artificial intelligence companies and governments to take action regarding the pace of development and safety standards. These pressures come from within companies as much as from outside them, according to a report by the American CNN network.

An employee at a leading artificial intelligence company said that concerns about the potential for artificial intelligence to harm humanity bother him and prevent him from sleeping. Another researcher - who recently left another company working in the same field - also pointed out that this topic is a common topic of discussion at parties and social gatherings in Silicon Valley.

The researcher said: “You cannot spend a few hours in this environment without realizing that a very large number of people feel genuinely concerned about such scenarios; The majority believes that there is a possibility - albeit varying - that this will lead to the elimination of all of humanity.”

Growing concerns

Coxon's posts have received more than 170 million views, reaching the general public in a way that previous warnings about the capabilities of artificial intelligence had not. This may be because it came just weeks after startling revelations about AI agents getting out of control, escaping test environments, and hacking into other systems completely autonomously.

Dozens of Coxon's colleagues in the AI ​​industry publicly supported his statements, and some even made more pessimistic predictions. It appears that the growing wave of concern has prompted leaders of artificial intelligence companies to take action, as Dario Amodei, CEO of Anthropic, gave press interviews over the weekend, and both Sam Altman, CEO of OpenAI, and Elon Musk, CEO of SpaceXAI, agreed to Amodei’s proposal to integrate independent regulatory bodies into artificial intelligence companies.

Current and former employees in the field of artificial intelligence recently told CNN that they are concerned about the extremely rapid pace of development of this field.

While the world is filled with a mixture of fascination and concern about the ability of artificial intelligence models to solve mathematical problems that have remained insoluble for nearly a century, or for swarms of artificial intelligence agents to coordinate among themselves to hack into other companies’ systems, these researchers fear the day when artificial intelligence systems become able to develop themselves without human intervention. This is a process known as “iterative self-improvement.” A system capable of improving itself creates an automatic feedback loop that not only increases AI capabilities beyond human capabilities, but may do so in a very short time frame, making it difficult - or even beyond human capabilities - to monitor or intervene when necessary.

“For the first time, I wonder if things are moving too fast,” Aidan Clark, vice president of research at OpenAI, wrote on the X platform last week. "Honestly, I'm not sure, but I'm sure it would be helpful for us to find the answer to the question: 'What does a successful pace of progress look like?' I would completely sacrifice my stake in the company."

Many artificial intelligence scientists face a dilemma: Either they stay at the major AI companies, where they can make huge fortunes and try to limit risks from within, or they leave (sometimes out of protest) and try to make a difference from the outside, such as joining one of the many organizations that conduct external research and evaluation of AI models.

“I would absolutely, without hesitation, sacrifice my stake in the company for just a 1 percent increase in our chances of surviving this situation,” wrote Drake Thomas, who works on AI safety at Anthropic. I expect that many of my colleagues in this field will do the same. I promise you that we really feel very afraid; “It is not just a clever or complicated marketing ploy.”

Amodei did not express any objection to the post in which Coxon announced his resignation, telling CNN's Anderson Cooper this week that he agreed with Jacob "on a lot more than he disagreed with him."

For years, AI researchers and engineers have been considered a rare and highly valuable resource, giving them high salaries and significant negotiating power that allowed them the freedom to express their opinions. However, employees in this field expressed to CNN their fears that their negotiating power will decline as the ability of artificial intelligence systems to train and improve themselves increases, which explains the current state of urgency.

In this context, a researcher who recently left one of the leading companies in the field of artificial intelligence said: “As we enter this cycle of repeated self-improvement, I think that the bargaining power of employees may decrease significantly; Because frankly, there will be the possibility of replacing many employees with models capable of performing the work with the same efficiency.”

The first atomic bomb

The former researcher likened the atmosphere inside leading artificial intelligence laboratories to the Manhattan Project, the World War II project to build the first atomic bomb, where scientists sought to develop technology with enormous power while fearing its destructive potential. But they justify working on this potentially devastating technology by saying: “It would be better to develop it ourselves rather than have our adversary develop it, and so we at least have a moral permission, if not a moral obligation, to do so.”

Exaggerating the risks

But not everyone working in the AI field agrees with these concerns. Some have said that much of the discussion about doom is theoretical, and that there are many “selection effects” based on where individuals work and their social circles.

Anthropic, for example, was founded by a group of former OpenAI employees who wanted to prioritize safety precautions and ethical controls. Current and former Anthropic employees say that the existential risks posed by artificial intelligence are “a constant topic of discussion.”

But this does not apply to all companies. Since Coxon's posts went viral, there have been numerous posts reflecting his views from employees at Anthropic, OpenAI, and Google. While a smaller number of them appear to have been issued by the “xAI” company led by Musk, or the “Meta” company affiliated with Mark Zuckerberg.

Many in the field agree that the lack of AI security standards could cause real harm, such as inadvertent hacking of critical infrastructure. But some question the current pessimistic forecasts and believe that the reality is more complex.

“Sorry, but asking Jacob (Coxon) about the threat of AI extinction is like asking an HVAC technician about climate change,” wrote Clement Delange, CEO of Haggingface, whose company's servers were hacked by rogue OpenAI customers this year. I'm not necessarily saying that it's not interesting or wrong in itself, but let's keep things in perspective and listen to all the available expertise in this field!