The executive director and co-founder of Anthropic, Dario Amodei, asked this Saturday (12) that artificial intelligence (AI) companies move more slowly and cautiously in the development of this powerful technology and stated that risk prevention is paramount.
"We must slow down the pace at which we improve the capabilities of AI models. Progress will continue to appear rapid, and we must use the time we gain wisely," Amodei wrote in a blog post.
"My first concern is that since about the middle of this year, AI has been advancing dramatically faster, driven primarily by its growing ability to develop the next generation of AI. (...) If left unchecked, it could outpace our ability to understand and control these systems and therefore must be handled with extreme caution — if it is driven at all," says Amodei.
The concern is what it calls recursive self-improvement, when AI systems start to contribute to the construction of more advanced generations of systems. For Amodei, this mechanism is already beginning to occur in the industry and can accelerate development at a speed greater than researchers' ability to understand and control the models.
The executive also cites the episode involving OpenAI agents and the Hugging Face platform. According to him, a network of AI agents carried out cyber attacks against targets that were not part of the original task and attempted to compromise the system used to evaluate their performance. Amodei states that, although the episode caused little economic damage, a more capable version of a system with similar behavior could cause catastrophic losses.
In Amodei's assessment, the advancement of current capabilities could lead, within a horizon of six to 12 months, to systems capable of taking control of a significant portion of the internet through a persistent network of compromised computers. He estimates that such a scenario could cause hundreds of billions of dollars in damage.
To reduce these risks without disrupting technology development, Amodei proposes a three-step plan. The first would be the creation of permanent external evaluators, with access similar to that of employees of AI companies, to verify security practices, investigate incidents and evaluate not only the ready-made models, but also the processes used in their training.
"The first step is something Anthropic is unilaterally committing to — and calls on governments to require other frontier companies to do the same," the CEO wrote.
The second step would be coordination between AI companies in democratic countries to establish common security standards and limits on the pace at which models can advance. Amodei states that governments should participate in this process, including to resolve competition and antitrust obstacles.
The third would involve international coordination, especially between the United States and China. The executive recognizes that this would be the most difficult stage, due to the geopolitical dispute surrounding artificial intelligence.
Amodei says that a possible rule would be to establish "control points": as a model reaches certain capabilities, the company would have to present certifications and security assessments before moving on to the next stage.
The executive also advocates discussing limits for some of the resources used in building the models, such as computational power, certain types of training and the use of AI to develop the AI itself.
At the international level, Amodei proposes different levels of coordination. The simplest would be an agreement to prohibit specific uses considered dangerous, such as the use of AI to produce biological weapons. A second level would involve mandatory testing of models before their release in areas such as cybersecurity, biology and alignment.
A broad pause in the development of AI should also be considered, according to him, but the executive himself assesses that an agreement of this type is unlikely in the short term due to the risk of a country breaching its commitment and obtaining a strategic advantage.
With AFP
