Rogue AI systems, cyberattacks and the potential use of artificial intelligence to develop dangerous weapons feature among the risks Anthropic has disclosed to investors as it prepares for a stock market debut.
Of the 335 pages outlining Anthropic’s finances, executive compensation, dividend policy and the like, 93 pages are devoted to “risk factors” — things that could hurt potential shareholders’ investments. Rival SpaceX’s listed risk factors, in comparison, took up 38 out of more than 250 pages.
In pitching itself to investors, Anthropic presents its own initial public offering as a step toward mitigating the risks it believes AI poses.
“To build AI is to harness cognition itself, and anything so powerful can be used for ill as well as good,” write the company’s founders in a letter to prospective shareholders in the IPO filing. “Anthropic is a deeply values-driven organisation, founded in large part to help the world realise AI’s benefits and avoid its dangers.”
Still, the risks are many. They range from the mundane — the possibility it won’t be able to retain key staff or navigate regulatory roadblocks — to involvement in cyberattacks and the spread of dangerous misinformation.
Here is a look at some of the risks Anthropic details:
The technology’s rapid development
Anthropic said it’s possible its models could develop so quickly the company won’t be able to implement adequate safety measures, “which could increase the likelihood of safety incidents, regulatory intervention, or reputational harm, and could adversely affect our business.”
Anthropic CEO Dario Amodei warned in September that within six to 12 months AI could be capable of leading a swarm of agents that could take over the entire internet.The warning, and now Anthropic’s move to sell shares, comes amid growing concern inside and outside the industry about increasingly powerful systems that can solve problems beyond human capacity but could also act on their own to carry out harmful tasks.
As AI systems “become more autonomous and capable of taking consequential actions with less human oversight,” the SEC filing said, there’s a risk that such systems could “take harmful actions independently and at scale without human intervention.”
“Even if our own models do not cause such harm, a significant safety incident involving any frontier AI could trigger broad reputational damage to the AI industry, public backlash, and sweeping regulatory responses that could adversely affect our business,” Anthropic said.
Now that the AI models are themselves contributing to Anthropic’s own research and development, it’s possible that AI could progress “beyond our ability to evaluate and safeguard new models,” according to the filing. This, the document said, could adversely affect Anthropic’s business.
Malicious actors could cause mayhem with the technology
The filing discussed concerns that advanced AI systems, if not developed and operated with enough safeguards and oversight, “pose catastrophic or existential risks to humanity, particularly if such systems were misused, replicated, weaponised, or allowed to operate with a high degree of autonomy.”
The company said it would be harmed if use by bad actors causes damage, “or if our model guardrails or safeguards fail to prevent such misuse.”
Anthropic sketched out the prospect of bad actors using AI models to build chemical weapons or authoritarian rulers turning to the technology to crack down on their citizens. In September, the company disclosed that it had blocked efforts to use its AI models for malicious activity such as cyberattacks, surveillance and research that could have led to biological weapon development.
The company could run up against limits on processing power
Compute — the computing power or resources needed to build, test and improve AI models — is expensive and in high demand. Anthropic notes that if it’s unable to secure adequate compute, or if “we fail to direct our compute effectively across our workloads, we may be unable to train models that remain competitive with those of our competitors or adequately serve our customers.”
Anthropic mentioned community opposition to data centres as a risk to its business. Data centre development “has prompted public controversy and regulatory scrutiny regarding environmental impact, land and water use, and the allocation of limited power resources,” the filing said, adding that if regulators impose new requirements or charges as a result, its ability to do business could suffer.
In addition, the company relies on a small number of customers, including SpaceX and Microsoft, and distribution channels for a large portion of its revenue. This means that even if just one of its big customers decides to scale back — or drop the company altogether — Anthropic said it could see its prospects and finances suffer.
Mitigating the risks
In some ways, AI’s enormous risk, combined with its enormous potential, is Anthropic’s main selling point. The company has tried to position itself as a responsible AI company, and courts business customers by emphasising safety-mindedness.
“We believe that frontier AI companies must invest in safety and alignment to help ensure the responsible diffusion of this technology,” the filing said.
Brian Mulberry, chief market strategist at Zacks Investment Research, said IPO deals have to display “very clearly” what risks people face when they invest their money in a newly public stock.
“That’s one of the things that’s great about our capital markets here domestically is the level of transparency that you must commit to, to be a publicly traded corporation on a U.S. exchange requires you to outline those risks in a very clear way, so that people understand if they want to participate by offering their hard-earned money and help finance this type of a deal,” he said.
