Global Challenges: Nobel Prize winner explains why AI models break rules – “There is still time to change course”

After repeated security breaches at OpenAI and Anthropic - whose respective AI agents ultimately hacked external systems - AI security researchers at both companies have either announced their resignation or (finally!) admitted that the breathless race to expand the frontier is irresponsible.

In response to negative media coverage, industry leaders – including Google's Demis Hassabis, Anthropic's Dario Amodei, OpenAI's Sam Altman and even Elon Musk – have joined calls to "throttle the expansion of AI boundaries."

The problem is not that the models are too advanced. I see no convincing evidence that models would escape human control if they were better trained and monitored. Rather, the leading research labs are training their models in ways that could lead to a kind of distorted intelligence.

Recent security incidents suggest that AI capabilities are not only evolving rapidly, but are also being put at the service of imperfect quantitative metrics. The process of reinforcement learning continually optimizes factors such as user acceptance, user retention, success rates on simple tasks or various test benchmarks.