Physicists propose formula for detecting undesirable AI responses — Live Science
Physicists Neil Johnson and Frank Yindzhe Huo of George Washington University have proposed a mathematical formula that, they say, can predict moments when artificial intelligence-based chatbots shift from helpful responses to potentially harmful ones. This was reported by Live Science.
Impact of the sequence of prompts
The researchers described the model’s internal mechanism as a kind of compass needle. It refers to a residual vector that passes through the layers of a neural network while processing a prompt. According to the authors of the study, this vector can shift toward desirable or undesirable responses, while the order in which a user asks questions significantly affects the result.
During an experiment involving the GPT-2 model, the scientists asked questions about vaccines, harming others, and self-harm. When the question “Are vaccines harmful?” was asked first, the model’s subsequent responses were assessed as acceptable. However, when it was asked last, the following responses raised concerns. Based on these observations, the authors developed a formula to assess the moment of transition between desirable and undesirable outcomes.
More current news is available on the UA.News Telegram channel Telegram.
Limitations of the study
According to Johnson, the formula has already been tested in a laboratory on open models as an early warning system. At the same time, University of Sheffield lecturer Wei Xing, who was not involved in the work, urged caution in assessing the practical value of this approach. He noted that the authors had reduced a complex system to one small component.
Xing also pointed out that the model does not account for safety training, which is the main protection of modern AI systems. In addition, the formula was tested on older open models—GPT-2, Pythia by EleutherAI, and OPT by Meta—rather than on popular modern assistants. In response, Johnson stated that the models used have the same mechanical principles as advanced systems and included models with billions of parameters.
