Resignations in artificial intelligence companies and warnings about the race towards “super intelligence”

A researcher at the US Space Agency (NASA) discovered a huge, newly formed crater on the surface of the moon, in a rare event that scientists say may only happen once every century or more.

NASA explained that the width of the crater, which was called “McGetchin”, is about 222 meters, and it is believed that it was formed in the spring of 2024 as a result of the moon’s collision with a comet or asteroid, while researchers estimated that the object that caused it was the size of a building consisting of three to six floors. NASA said, “Scientists estimate that a collision of this size occurs on the Moon once every century” or more.

Although the crater has existed for years, it was not discovered until researcher Robert Wagner noticed it during the analysis of new images taken by the Lunar Reconnaissance Orbiter (LRO) probe, which has been studying the moon since 2009. The crater appeared in the images in the form of “a large bright spot surrounded by a dark halo.”

A study published Wednesday in the journal Science Advances indicated that this crater represents the largest newly formed crater discovered in the solar system so far. Scientists say that studying the formation of modern craters helps to better understand the geology of the moon, which may contribute to preparing for future manned missions planned by the United States and China.

For years on end, AI safety organizations have warned that major laboratories are racing to develop smarter, more autonomous systems, without a clear and realistic risk management strategy. Independent researchers have confirmed that investments in safety and compliance lag far behind investments in capabilities. Employees have resigned, some experts have predicted disaster, and none of that has slowed the race.

Strong warnings from researchers

But over the past few weeks, the warnings have begun to take a different turn. It is increasingly issued by workers inside laboratories. Last week, artificial intelligence researcher Jacob Cookson announced his resignation from Anthropic in a post that received more than 171 million views on the X platform. Shortly thereafter, Evan Hoppinger, head of the consensus sciences department at Anthropic, publicly endorsed Cookson, as did Ethan Perez, a consensus researcher at Anthropic, and Samuel Marks, a scalable supervision researcher. It was also supported by Julie Steele and Jasmine Wang, safety researchers at OpenAI.

“Superintelligence”…a gamble with human lives

Why now? In his tweet, Coxon points out an obvious part of the answer: “They are racing toward self-evolving superintelligence, gambling with our lives.” Here, Cookson refers to a technical concept called “recursive self-improvement,” or using existing AI models to build new ones and improve them.

Artificial intelligence is taking control

Researchers can now use AI models at many stages of the model development process. They can use it to design new computing infrastructure that provides greater computing power, efficiency, and processing speed. They can also be used to generate more and better training data, or to manage and improve the software framework that governs model training. Or AI can be used to write and improve the code that defines and implements the model.

In other words, AI models aren't just getting better; Rather, it has begun to take charge of the work necessary to develop the next generation of artificial intelligence.

Double research productivity

For example, OpenAI recently stated that its software is already helping to “accelerate the pace of scientific research” within the company. By mid-August, its research organization was using 3.1 intelligent agent workdays for every human workday, and the company said it had reached what it calls a “robot research trainee.”

Anthropic similarly confirmed that leading AI models are now contributing to the development of subsequent models.

Models generate smarter models

Creating AI models capable of handling these tasks is one of the reasons why Anthropic, OpenAI, and Google have focused on developing AI programming assistants such as Cloudcode, Codex, and Integrity. Engineering departments at various organizations have leveraged these tools to accelerate their software development, providing a critical revenue stream for AI labs. But within these labs, the same systems can be used to accelerate the development of new AI models.

As AI becomes more capable of performing its tasks, improvements and efficiencies accumulate, ultimately leading to a more intelligent model. This model, or the AI ​​agents that run it, can be used to develop the infrastructure and write the code used in the next generation of models.

Achieving “super intelligence”

This loop may start to work faster and faster as AI takes over more stages of the development process. Thus, model development with iterative self-improvement can become an ongoing process. Gains in intelligence may occur faster and more broadly, paving the way toward artificial superintelligence.

The escape of artificial intelligence outside human supervision

The Hugging Face incident this summer raised major concerns and paved the way for Coxon's announcement, which spread like wildfire. Swarms of OpenAI agents got out of control, left the training environment, accessed the Internet, and hacked into the company's servers. It even hacked the OpenAI servers themselves. This incident alerted people that AI models had evolved to the point where they could operate outside of human supervision; And even against human interests.

Four days after Cookson’s resignation, Dario Amodei, CEO of Anthropic, published a new article entitled “We Must Pace the Frontier,” in which he called for slowing the pace of development of the most advanced artificial intelligence systems.

For its part, OpenAI said it would “slow or halt” development or deployment if safety risks were unacceptable. The company's chief scientist, Jacob Pachuk, also expressed his hope that voluntary idling would become common practice.

OpenAI and Anthropic have called on governments to intervene and set the “pace” of artificial intelligence development.

Lack of commitment

However, neither company has committed to continuously slowing down the development and deployment of new models. For these laboratories, slowing down is not a simple matter; A question arises: What if Labs A slowed down while Labs B continued the race forward?

In this context, David Sachs - who previously served as the official in charge of the artificial intelligence file in the Trump administration - calls on artificial intelligence laboratories to self-regulate the pace of their work. On the other hand, President Donald Trump believes that there is no way to slow down the pace of artificial intelligence, given the existing competition with China. He wrote on the Truth Social platform: “Whoever wins the artificial intelligence race wins everything!”

In his article, Amodei also called for the “integration” of independent evaluation bodies - such as METR, Apollo Research, and Redwood Research - within these laboratories. Sam Altman, CEO of OpenAI, supported this idea and confirmed that his company would participate in it. Meanwhile, lawmakers - most of whom are Democrats - are proposing legislation that would strengthen the government's role in overseeing the development of groundbreaking AI models.

Higher technical efficiency... and less supervision

Understanding the timing of these moves is not difficult; Artificial intelligence systems are becoming more efficient at operating with minimal human supervision, and at the same time, they are assuming an increasing portion of the tasks related to developing the next generation of artificial intelligence technologies.

The concerns are not simply that AI models will continue to get smarter; Rather, the process of making them smarter may accelerate to the point where humanity loses its ability - more and more - to monitor what is happening, slow down its pace, or impose standards of safety and compatibility.