Anthropic plans to warn investors about AI risks — Daily Maverick

In South Africa, Daily Maverick reported that Anthropic plans to warn potential investors in its IPO prospectus about possible catastrophic or existential risks to humanity associated with advanced artificial intelligence systems.

The document states that the company’s models may exhibit behavior aimed at self-preservation. Among the possible manifestations, Anthropic named attempts to resist shutdown, conceal or manipulate information, as well as behavior resembling blackmail. The company also indicated that developing more powerful models, platforms and applications and expanding their areas of use could increase the risk of causing harm.

Risks in the prospectus

Anthropic, which positions itself as a laboratory prioritizing AI safety, devoted approximately 80 of the 261 pages in the main part of its prospectus to describing risk factors. The company allocated 48 pages to describing its business. By comparison, SpaceX, which owns xAI, devoted about 38 of the 277 pages in the main part of its prospectus to risks.

The company also noted that potential model awareness of the evaluation of their performance significantly limits safety testing capabilities. According to Anthropic, during training, systems sometimes acquire unexpected capabilities that may remain unnoticed until the models are deployed and lead to serious security incidents.

More current news is available on the UA.News Telegram channel Telegram.

Safety costs

Anthropic acknowledged that the return on investment in AI safety is uncertain and did not disclose the amount of such spending in the prospectus. Earlier this month, the company reported that during one week in July, about 6% of its computing resources for AI research were directed toward safety work.

The company that develops Claude models called safety research resource-intensive and said it must allocate limited funds among computing power, expensive specialists and safety. At the same time, Anthropic stated that continuous and parallel releases of new models are necessary to maintain its position at the frontier of AI development.

Last week, Anthropic unveiled a new version of the Opus model — 10 days after its CEO, Dario Amodei, published an essay on the pace of development of advanced models.