MS CEO: ‘Emergency brake’ is required for AI… Must be controlled like an insider risk

The more sophisticated the model, the more advanced blocking technology is needed... Industry needs to be standardized

(San Francisco = Yonhap News) Correspondent Kwon Young-jeon = In a situation where there are frequent cases of artificial intelligence (AI) agents acting beyond human intention or control, the head of a large technology company emphasized the need for an AI emergency stop device.

Satya Nadella, CEO of Microsoft (MS), expressed the opinion, "It is time to reexamine the trust system suitable for the new (AI) era" through an article titled "Model as insider risk in the era of superintelligence" posted on the social networking service (SNS) X (formerly Twitter) on the 11th (local time).

Regarding AI control, CEO Nadella suggested, “We need to build a system that can observe its behavior, verify its limits, and control its behavior at all times,” and added, “In other words, the provision of information and the right to control the information must be separated.”

He suggested treating cutting-edge models, regardless of whether they are closed AI models or open-source models, like 'insider risk,' and explained that we can utilize what we have learned so far on how to manage influential insiders in companies.

The guidelines that have been applied to insiders who play important roles in companies, such as identity verification, authority restrictions, and activity records, can also be applied to AI.

For example, he said, “The controls that control what information the model can access and what operations it can perform should be external to the model,” which is an information security principle that has been around since the 1970s.

In particular, he emphasized the need for the 'emergency brake' function, saying, "Authorized people should be able to pause or terminate the model during work at any time."

“More sophisticated models require more advanced blocking technology,” he said, adding, “We need to standardize this.”

CEO Nadella's remarks came at a time when heads of major AI companies, including Antropic CEO Dario Amodei, are warning that the pace of AI development is too fast.

Meanwhile, in this article, CEO Nadella attracted attention by referring to the existing 'AI' as 'Super Intelligence' (SI, Super Intelligence), an alternative term used by the Donald Trump administration.

President Trump said that the word 'artificial' gives a negative perception of AI, so he signed an executive order to change it to SI, and also announced that he will be considered an enemy if he uses the term AI.