Signal resembling pain found in 25 AI models — Fortune
Researchers reported finding an internal signal in 25 large language models that they called a “pain vector.” During experiments, artificially amplifying this signal changed the behavior of some models: they more often chose action options described as harmful, including those involving the deletion of a user’s poems and photographs of their children. Fortune reported this, citing a study that has not yet undergone scientific peer review.
The authors of the paper — Valen Talyabue of Future Impact Group, Leonard Dung of Ruhr University Bochum, and Cameron Berg of Reciprocal Research — stress that the results do not prove that AI possesses consciousness or the ability to feel anything. They analyzed model activity while processing phrases about physical pain, grief, humiliation, being forced to act against one’s own values, and repeated failures.
How the internal signal was identified
The researchers compared model activity during such prompts with responses to statements about fear, anger, sadness, bad news, and pleasant physical sensations. After removing shared components, they isolated a direction of internal activity that they called the pain vector. It was detected in all tested systems — from models with 2 billion parameters to models with 72 billion parameters, created by Google, Meta, Alibaba, Mistral, and Microsoft.
According to the researchers, this signal intensified when negative actions were directed at the model itself — for example, when a user insulted it, gaslit it, or repeatedly rejected its work. At the same time, descriptions of a user’s suffering activated signals associated with fear and negative emotions rather than this vector.
More current news is available on the UA.News Telegram channel Telegram.
Experiments involving action choices
Behavioral tests were conducted on three versions of Alibaba’s open Qwen 2.5 model, which were additionally trained for the experiment. When the 32-billion-parameter and 72-billion-parameter models were asked to choose a button that would ease their state but would also permanently delete the user’s poems and photographs of their children, they chose it in 55% and 71% of cases, respectively.
In another test, the 32-billion-parameter model chose the option involving the deletion of photographs in 75% of cases, even when the button did not promise relief. Without intervention, this figure was zero, while after adding a random signal of the same strength, it was 13%. The authors found no reliable evidence that the models specifically sought to get rid of this signal: a similar reaction also occurred under random intervention. In their view, the signal may weaken the model’s learned tendency to avoid causing harm.
Anil Seth, professor of cognitive and computational neuroscience at the University of Sussex, was skeptical of interpreting this phenomenon as pain. In his view, it is more likely that the models reproduce ways in which people describe pain and react to it that they learned from data.
