AI researchers warn of risks from self-improving systems — Jerusalem Post
Israel’s Jerusalem Post reports that current and former employees of OpenAI and Google DeepMind warn that companies are not doing enough to protect the world from the potentially catastrophic consequences of creating artificial intelligence systems capable of improving themselves and surpassing humanity’s ability to control them.
Researchers’ appeal
The concerns of scientists were compiled by Palisade Research, a nonprofit organization focused on AI safety. Video testimonies from participants in the frominside.ai project were provided to Reuters. The researchers said their concerns about existential risks are sincere rather than a marketing tool. They also argue that AI laboratories reward employees who create new models more than those who call for caution.
Geoffrey Irving, co-founder and chief scientist of the nonprofit organization Resolution, who worked at OpenAI and DeepMind, told Reuters that risks are growing quite rapidly. In his view, industry experts should speak directly about the dangers.
Concerns over losing control
DeepMind research scientist Neel Nanda said in a video that he estimates the probability of humanity’s extinction because of AI at no less than 10% and called such a figure extremely high. OpenAI AI alignment research engineer Juan Felipe Ceron Uribe said leading laboratories are competing with one another without acting cautiously enough.
More current news is available on the UA.News Telegram channel Telegram.
Some current and former researchers believe society is not ready for the next generations of models, especially if they gain the ability for recursive self-improvement — continuous learning and acquiring new capabilities with almost no human involvement. Former OpenAI policy researcher Rosie Campbell said that before she left the company in 2024, the organization was becoming increasingly isolated, and it was becoming more difficult to influence the direction of technological development.
Calls to slow development
Anthropic CEO Dario Amodei published an essay this month calling on the industry to slow AI development. OpenAI CEO Sam Altman supported this position. At the same time, both companies released new models while competing for customers, although OpenAI said it had not released an even more powerful model.
Former OpenAI governance researcher Daniel Kokotajlo said that after the incident involving Hugging Face, former colleagues privately approached him with their concerns. According to him, laboratory leaders are convinced that unilaterally slowing their work could make the situation worse.
