OpenAI Chief Scientist Warns of AI's Runaway Progress, Calls for Extreme Caution

OpenAI's chief scientist Jakub Pachocki has called for "extreme caution" over AI's runaway progress and warned more intervention may be needed to ensure "humans remain in control of the future".

"I am concerned no one is prepared for the consequences of a continued rapid rise in machine intelligence," he wrote in a blog post entitled "An Alien Mind, external".

The post comes only a few days after the ChatGPT-maker released its latest model GPT-6 Astra, which it called its most powerful product ever.

OpenAI and other AI firms like Anthropic shared reports of their AI agents acting autonomously and carrying out real-world cyber-attacks on other companies.

In July, OpenAI called an incident in which its AI agents - AI systems which can operate alone after human instruction - hacked the tech platform Hugging Face "unprecedented".

Meanwhile in September a report claimed months earlier AI agents from the firm had also hijacked a German website.

"We are facing a transition to a world with incredibly intelligent machines, and we need to ensure that transition works out well for humanity," Pachocki wrote in the latest blog post.

He said OpenAI would continue to "build defensive systems" and seek technical solutions to alignment - a term used to describe ensuring a machine's actions and goals perfectly match human intent and safety guardrails.

Pachocki added one of the firm's main priorities moving forward would be to build an "automated AI researcher" to keep pace with AI progress while ensuring ways that human researchers could remain part of the process.

"Instead of better AI guardrails, regulations, or assurance to keep people safe, they propose developing internal AI agents to research these problems," said Professor Gina Neff, head of the Minderoo Centre for Technology and Democracy at the University of Cambridge.

"Such answers to growing concerns about the problems OpenAI's models are causing for cyber-security, job loss, mistakes, errors and fraud are simply not good enough."

Nathan Calvin, general counsel at the advocacy group Encode AI, said he agreed with Pachocki on the hazards present in advanced AI model development - but he claimed OpenAI was unwilling to be transparent, and said this meant warnings risked being dismissed as "just self-interested hype".