Artificial intelligence brake recommendation from Anthropic and OpenAI

Anthropic and OpenAI have proposed different methods to the California government to stop dangerous AI systems. Anthropic wants the state's attorney general to be given the authority to intervene through the courts. In contrast, OpenAI advocates emergency shutdown mechanisms that developers are required to both install and use when necessary.

Both companies forwarded their recommendations to Governor Gavin Newsom. Because Newsom signed executive order N-9-26 on September 18. This document asks government agencies to prepare additional security recommendations for the most powerful AI systems. The deadline for proposals is November 16. However, the decree does not currently impose a new obligation.

First Difference Between Anthropic and OpenAI: Anthropic's Methodology

In its Oct. 15 document, Anthropic recommended independent evaluation of developers' security claims. He also asked for the results to be published. Among the demands is the obligation for companies to publish a risk report at least every six months. Moreover, if the models used in-house are significantly stronger than the publicly available versions, their risks should also be included in the report.

Independent evaluators should have access to the strongest models and insider knowledge of developers. However, the studies should be financed by a common fund managed by the state, not by direct payments from the inspected companies.

Anthropic also introduced the concept of the circuit breaker, meaning "automatic fuse". These mechanisms should limit the problem to the level of a single agent, customer, deployment environment, or entire model. If an independent review confirms the risk of catastrophic harm, the attorney general should be able to apply to the court to temporarily suspend the system.

OpenAI's Different Approach to Anthropic and OpenAI Recommendations

OpenAI supported the independent technical evaluation in its October 20 letter. But he suggested separating this from compliance monitoring. On the other hand, technical reviews will look at evidence of the safety of models and the impact of protection measures. The audit will check whether the rules and legal obligations published by the company are complied with.

The company wants the independent audit to be conducted annually. Technical inspections should be carried out when serious risks arise during the training, testing and internal use stages. At the same time, OpenAI asked California to develop joint evaluation standards with the US National Institute of Standards and Technology (NIST).

There is also a difference of opinion regarding the method of closure. According to OpenAI, the term "emergency shutdown mechanisms" is more accurate than the term "kill switch". Because there is no single button for every artificial intelligence system. Depending on the situation, model training may be stopped, an agent may be isolated, vehicle or network access may be interrupted, or the service may be suspended.

Responsibilities should also be shared among developers, IT service providers, and companies using the models based on the processes they actually manage. However, the developer cannot always shut down independent copies of vulnerability-heavy models running on third-party infrastructure.

Anthropic and OpenAI agree on expanding event reporting. Models exceeding protection limits or acting outside their authority should also be reported. According to Greyhound Research analyst Sanchit Vir Gogia, technical mechanisms should enable the operator to quickly limit dangerous behavior. The independent audit should examine not only the measures taken, but also the developer's failure to act. The deadline for proposals is November 16.