Microsoft's Nadella says AI needs an ‘emergency brake’ that humans control

Microsoft Chairman and CEO Satya Nadella on Saturday called for advanced artificial intelligence systems to be built with containment, independent controls and an "emergency brake" that lets authorized people pause or shut a model down mid-task.

"We need to surround non-deterministic models with strong, deterministic system design, human controls, and reliable operating procedures, and establish industry standards where existing ones are insufficient," Nadella wrote in a post on social media platform X.

"Treating frontier closed and open weight models like insider risks is a way to build such a system," he said.

Nadella's comments come amid warnings from top tech executives and researchers — including Microsoft co-founder Bill Gates, Anthropic CEO Dario Amodei, OpenAI CEO Sam Altman and SpaceX CEO Elon Musk — about insufficient AI safety protocols and claims that the technology is moving too fast.

An AI researcher quit Anthropic last month and accused the company, and its chief rival, OpenAI, of "gambling with our lives." Later that day, an alignment lead at Anthropic focused on AI safety said there's a greater than 10% chance that the technology could "kill all humans" within the next decade.

Conversely, President Donald Trump has repeatedly dismissed AI extinction risks and has instead emphasized that the industry needs to stay ahead of China. Trump also recently introduced a new "AI Force," led by Director of National Intelligence Jay Clayton, to help facilitate the industry and root out bad actors.

Microsoft's Nadella wrote Saturday that AI systems should be designed around "principles of observability."

They include model diversity; a human-readable footprint of the model's actions; continuous system testing; independent controls and auditability; containment; and incident disclosure, Nadella wrote.

"The most trustworthy Super Intelligence system will not be the one with the model we trust most," he wrote. "It will be the one that enables us to trust the model the least."