As researchers warn that artificial intelligence could pose catastrophic risks, some of the pioneers who helped develop the technology are weighing in with additional warnings on the dangers of unfettered advances in capability.
A series of stark AI safety warnings has emerged over the past week, with researchers from major AI labs like Anthropic, OpenAI and Google DeepMind all sounding the alarm about potentially catastrophic risks from developing the technology.
AI bosses including Sam Altman, Dario Amodei, Demis Hassabis and Elon Musk have shown rare agreement in calling for a slowdown and regulatory oversight.
Meanwhile, concerns have also intensified debate in Washington. While some members of Congress from both the Democratic and Republican parties have advocated for bills to check the progress of AI, U.S. President Donald Trump on Monday raged against growing calls for AI regulation.
Here's what the people who helped create AI think of the existential risks:
Yoshua Bengio
Yoshua Bengio is a Canadian computer scientist, considered to be a central pioneer of modern deep learning.
"Scientists at frontier AI labs have unique insight into the most advanced models," Bengio said on Sept. 9, in response to Anthropic researcher Jacob Coxon announcing his resignation. Coxon had warned that AI could "kill us all by the end of the decade."
Scientists often see "the associated risks months before models are released to the public," he added. "Their perspective is vital for keeping society informed and should be taken very seriously."
Days before the social media storm started, OpenAI's chief scientists warned "no one is prepared for the consequences" of AI.
In a blog, published on Friday, Bengio said that today's AI systems "already have the necessary hacking skills and the powers of persuasion to be turned against human interests in seriously harmful ways."
Recent events, he added, have shown that AI can plan over days or weeks, and the risks would be "much worse if their ability to strategize over the long term continues to advance."
"My concern with AI companies' current attempts to mitigate misalignment is that these efforts may only hide it, by rewarding and selecting the AIs that cheat without getting caught," Bengio said. "We should certainly continue research toward better monitoring of AIs' actions, their chains of thought, and the activity inside their networks."
Geoffrey Hinton
Geoffrey Hinton, professor emeritus at the University of Toronto, is considered a pioneer of neural networks and deep learning.
When asked by the BBC on Sept. 10 whether he thought there was a greater than 10% chance AI could kill all humans within a decade, Hinton said: "This is the kind of thing that's very hard to estimate, because we've never had anything like this before. We've never created beings that may soon be smarter than us."
A 10% chance was not "unreasonable," he said, adding that "nobody really knows how to give a sensible estimate."
AI systems could design "very nasty" biological viruses, alongside computer viruses and commit "devastating cyber attacks. There's countless other ways it could get rid of us if it wanted to. We have to figure out how to design it so it won't want to," he said.
"We have two possible futures," Hinton added. "One where we figure out how to deal with those dangers before they get out of control, and the other where we don't."
Aidan Gomez
Aidan Gomez, co-founder and CEO of AI company Cohere, was one of the authors of the 2017 research paper "Attention Is All You Need," which helped lay the foundation for today's leading AI models.
"I think that these models are the most potent cyber weapon that has ever been created, that we've ever seen. They are incredible at finding and exploiting vulnerabilities at scale," Gomez told CNBC's "The Tech Download" podcast on Sept. 1.
"Recent incidents make one thing clear, the security of the deployment environment determines safety," he said, referring to cybersecurity attacks conducted by rogue AI models over the summer.
"Weak container security is what allows breakout events, not any inherent push toward autonomous systems," Gomez told CNBC. "As policymakers consider how to govern AI, they should recognize that different systems carry different risks, and rules shouldn't be shaped solely by big tech companies chasing AGI."
