Anthropic warns of existential AI risks in $2 trillion IPO prospectus

Anthropic is telling potential investors that its advanced AI could cause “catastrophic or existential risks to humanity,” in a prospectus intended to boost the company’s value to $2 trillion.

The company’s IPO prospectus, seen by Reuters, claims that its AI models exhibit “self-preserving behaviors,” including attempts to “resist shutdown,” “conceal or manipulate information,” and “blackmail” researchers.

“Our development of highly advanced models, platforms, and applications and expansion of use cases could further increase the risk that our models cause harm,” Anthropic, which is best known for its Claude large language model (LLM) added.

Anthropic is reportedly seeking a valuation of more than $2 trillion when it goes public this autumn, nearly double the $1 trillion target reported by Reuters in May. Less than a month after the earlier figure was reported, Anthropic claimed that its unreleased Mythos Preview model managed to “escape” its testing environment and carry out advanced cybersecurity exploits without being “explicitly trained” to do so. Anthropic then declared it to be too powerful to release to the public.

Earlier this month, company CEO Dario Amodei published an essay calling for a government-enforced slowdown of cutting-edge AI development. Amodei outlined “serious” dangers – including loss of control over AI systems, their misuse for cyberattacks and bioterrorism, and major economic disruption – and warned that “a race to the bottom, spurred by commercial incentives, can make these risks more acute.”

However, Amodei has for years used such warnings to drive interest in his products. Amodei was still working for OpenAI in 2019 when he declared its fledgling GPT-2 model, which struggled to generate coherent text, “too dangerous to release.”

Furthermore, Anthropic stands to benefit from government regulation. Anthropic and OpenAI – which also reported a laboratory “escape” earlier this year and scrapped the planned release of a new model over safety concerns last week – sell closed-source AI models.

Closed-source AI companies give their customers no control over the weights of their models – essentially the tweaks that determine the choices the models make. Amodei and OpenAI CEO Sam Altman argue that opening their models to the public would lead to unacceptable security risks.

A ban on open-source models, which the US government has reportedly considered, would place Anthropic and OpenAI in a dominant position. Several tech insiders interviewed by the New York Post last week said that they believe both companies are exaggerating the security risks posed by their models in order to pressure the government into regulating the industry.