Anthropic has issued the clearest warning yet about self-improving AI. The revised S-1A document published by the company for the public offering on Monday attracted attention not only with its income and expense figures. The risk sections of the document say that the threshold for recursive self-improvement (RSI) is being approached.
Anthropic self-improving artificial intelligence: What is RSI?
RSI describes the point at which AI has gotten so good at training itself. At this point, there is a sudden acceleration in artificial intelligence capabilities. However, this concept affects the two camps differently.
For the pessimistic segment, RSI means an irreversible disaster. Because artificial intelligence can get out of human control and remain indifferent to the damage it causes while achieving its goals. Optimists see it as the promise of efficiency and scientific discovery.
Anthropic, on the other hand, considers RSI as both an opportunity and a danger. The company says that it uses artificial intelligence in its R&D studies. Moreover, he adds, using AI to produce more capable AI could pose a serious security risk. The risk increases, especially if the rate of progress increases sharply.
Statements added to risk factors in the document
The company added new sentences to the Risk Factors section of the document. First, he admits that he did not realize the leaps in capability in some of the new models he developed with artificial intelligence at the time. He also states that it may soon reach the RSI threshold in its internal definition.
According to the document, internal forecasts and expectations for future models indicate that the Responsible Scaling Policy's definition of "dramatically accelerating progress due to artificial intelligence" may soon be met. However, none of the existing models are considered to exceed this threshold. The company therefore continues to evaluate the underlying data.
Anthropic also notes that acceleration can sometimes only be noticed after it has occurred. Because improving security, compliance and safety measures may be slower than the progress of the models.
Will human control continue in Anthropic self-improving artificial intelligence?
The company promises to keep people involved throughout the development process. Anthropic does not deem it safe to use Claude with full autonomy to design successive generations of advanced models without meaningful human control or insight. That's why it's considering updates to its Responsible Scaling Policy that will keep lead model development under human control.
No current pattern is considered to have crossed the RSI threshold.
The use of artificial intelligence in R&D is increasing.
Policy updates are being evaluated.
Skeptics, however, remind that the S-1 document serves both informational and marketing functions. In this respect, the claim of approaching a model that is strong enough to improve itself can also be read as a message to investors. Still, Claude's coding, math, and cybersecurity skills have improved rapidly this year.
Now all eyes are on the updates Anthropic is considering making to its Responsible Scaling Policy. However, the company has not yet shared the details of these changes. Uncertainty will continue until Anthropic announces the steps it will take to reach the threshold of self-improving artificial intelligence.
