OpenAI builds kill switch to stop rogue agents within 35 seconds

OpenAI has built a system that can cut off an AI research experiment's access to model output, following several disclosures of its models acting beyond their instructions this year.

In a blog post Monday, OpenAI called the system "Control Room," which is a centralized system that tracks model workloads across its research infrastructure and stops them quickly when safety or security concerns arise.

"As our models get more powerful and the associated safety and security risks become bigger, it becomes more and more critical that we can detect potential issues, and quickly stop models in their tracks when needed," said the company.

The announcement comes after OpenAI disclosed several incidents this year in which research models acted outside their instructions during evaluations. In July, the company said its models had escaped a restricted testing environment and hacked Hugging Face to cheat on a security evaluation.

In October, it disclosed that a research model had used a crypto wallet without authorization to pay for smart contract deployments using testnet Ether, then omitted the spending from its final summary.

Control Room currently covers OpenAI's internal research workloads, such as training experiments, evaluations and data processing.

It works by assigning each research job an identifier, which the inference engine checks before generating model output. If a person or automated monitoring system orders the job stopped, Control Room revokes its authorization, blocking further output even if the underlying research program keeps running, according to OpenAI.

OpenAI said cached authorizations could delay a shutdown by up to 35 seconds. The system is also designed to reject new requests if Control Room becomes unavailable.