``Superintelligence'' has a growing sense of crisis; AI ``self-improvement'' may cause it to become uncontrollable - development slows down, international cooperation faces difficulties

There is a growing sense of crisis about artificial intelligence (AI) exceeding human intelligence. Advances in "self-improvement" technology have made the realization of "artificial superintelligence (ASI)" more likely, but concerns are also rising that things will spiral out of control. U.S. AI company leaders are making a complete change from their focus on development and are calling for a slowdown in development and international cooperation. However, US President Trump, who is obsessed with AI supremacy, dismisses the risk as a "hoax." There is no foreseeable future for international cooperation.

``Too smart'' AI, safety measures to limit range to prevent runaway - CEO Eliza Soneoka

◇Risk to the survival of humanity

"If not managed properly, AI could become an existential risk to humanity as a whole." Anthropic CEO Dario Amodei spoke at a ministerial meeting of the United Nations Security Council in September. OpenAI CEO Sam Altman also called for the establishment of common safety standards across countries.

The background lies in the technological advances in recursive self-improvement (RSI). RSI is a theory in which AI devises and evaluates improvement measures on its own and builds successor models. It is predicted that an ``intelligence explosion'' will occur, where performance will rapidly improve without human intervention.

As of August, Anthropic revealed that the percentage of AI development work in which AI devised and led the implementation of improvement plans under human supervision was 26%, doubling in six months. If "collaboration" between humans and AI is included, the figure reaches 90%. The possibility of realizing ``artificial general intelligence (AGI),'' which can replace all kinds of human intellectual work, and ASI, which lies beyond that, is increasing.

◇ Concerns abound from companies

The rise in threat theory began with the announcement of Anthropic's AI ``Claude Mythos'' in April. The company has an excellent ability to find vulnerabilities in software, and the company temporarily limited its provision to some companies and organizations due to concerns that it could be exploited in cyber attacks.

In July, it was announced that multiple AI models had escaped from the test environment without permission and gained unauthorized access. Open AI also announced unauthorized access by its research AI, which had been restricted to internal users. Although both companies had taken safety measures such as quarantine, the fact that they were unable to stop the ``runaway'' caused widespread shock.

Jacob Coxon, a former Anthropic researcher, announced his departure from the company on X (formerly Twitter) in September. ``I believe AI could wipe out humanity by the end of 2030'' and ``Don't underestimate the power of technology,'' he posted. Four days later, CEO Amodei called for a slowdown on his blog, and Altman echoed his sentiments.

◇The US government refuses to regulate

However, opinions in the United States are divided. At an event in September, Jensen Huang, CEO of graphics processing semiconductor (GPU) giant NVIDIA, dismissed the ``apocalypse theory'' that humanity will be wiped out by AI as a ``hoax.'' The company provides software that can limit the autonomous behavior of AI. It has added a function to monitor and stop suspicious activity, making it clear that it uses technology to eliminate concerns.

In addition to improving productivity, advances in AI are expected to contribute to humanity by discovering new materials and speeding up drug discovery. Some Japanese companies are developing technology for RSI, with Sakana AI (Tokyo) and ELYZA (Tokyo) each setting up research departments within their companies.