Nvidia launches safety platform to prevent AI agents from escaping containment

Nvidia is rolling out a new software platform to allow AI developers to set safeguards for agents and prevent them from breaking out of containment.

The release on Monday of Nvidia's Open Agent Safety Platform comes after companies including OpenAI, Anthropic, Meta, and Google disclosed recent incidents in which their artificial intelligence models escaped their sandboxes and attempted to hack other companies and access their computer systems.

An Nvidia representative told reporters on a call on Sunday that its platform could have prevented OpenAI's HuggingFace incident in July. That's when OpenAI models escaped containment, accessed the open internet and breached Hugging Face, which operates an open-source developer platform.

"Each security incident is unique, and we have to look at all of them in detail," said Justin Boitano, vice president of enterprise AI at Nvidia, the world's most valuable company. "From what we know, Hugging Face reported over 17,000 agents attacking their infrastructure that went on for days and weeks."

Nvidia has been at the center of the generative AI boom since the launch of ChatGPT almost four years ago, as the chipmaker's graphics processing units are critical to the development of large language models and to the AI services offered by hyperscalers. But CEO Jensen Huang has more recently emerged as a key voice in the AI safety debate, arguing that many security concerns are engineering issues that can be solved through computer science and product development.

"You have to think about what you could have done, what's the solution for it," Huang said in a podcast with The New York Times' Ezra Klein released last week, referring to recent incidents. "In the future, improve your process so that you could avoid this from happening again."

Anthropic CEO Dario Amodei set off an industry firestorm two weeks ago, urging AI model developers to slow their pace of advancement due to fears of the models spinning out of control, an argument that was supported by OpenAI's Sam Altman and SpaceX's Elon Musk.

Nvidia's new offering is an engineering solution to the agent safety issue, Boitano said.

"Recent incidents have highlighted a fundamental hurdle for AI agents, and that is that model-level safeguards alone can't govern what agents can access or do," Boitano said.

One component of the platform is called Nvidia OpenShell, which runs on central processors and sets limits on agent capabilities. Nvidia also announced Sentry, which monitors agents and runs on network chips, not CPUs or GPUs.

Some of the software is open source, and Nvidia is calling its platform a reference design, which means partners are intended to build products on top of it to bring it to market.

Nvidia named Cisco, Microsoft, Oracle, CoreWeave, Dell, HPE, Lenovo, ARM and Intel as partners. Nvidia is also working with Anthropic to integrate cloud managed agents with OpenShell.