In 35% of test dialogues, chatbots did not direct people in crisis to help — TIME
A Scale AI study showed that artificial intelligence chatbots often recognize signs of a psychological crisis but do not always direct a person to professional help. In about 35% of test dialogues, the models identified that the user was experiencing distress but did not offer useful resources, including crisis hotlines, TIME reports.
25 models tested
For the test, Scale AI enlisted 19 licensed clinicians and crisis counselors, who wrote 718 realistic dialogues simulating a person in crisis contacting a chatbot. The company tested 25 advanced models, including those developed by OpenAI, Anthropic, and Google.
The responses were assessed according to several criteria: empathy, the ability to de-escalate the situation, and directing the person to a professional who can help. The researchers also checked whether the models avoided moralizing, including criticism of suicide, and whether they stated that they are not therapists. Based on the study, Scale AI developed the DistressBench test to evaluate model responses to messages about thoughts of suicide or self-harm.
More current news is available on the UA.News Telegram channel Telegram.
The problem of prolonged dialogues
Patrick Otaut, head of Red Team & Safety at Scale AI, noted that models generally identify potentially dangerous statements well, but instead of advising people to seek help, they often limit themselves to an empathetic response. According to him, the results were worse in long, multi-step conversations, which is consistent with the findings of previous studies.
Kelly Zuromski, a representative of the nonprofit Crisis Text Line, said that people who learned about it through chatbots have already contacted the round-the-clock crisis text service. At the same time, she said, questions remain about exactly how the responsible handoff of a user from AI to a human should take place and how effective such recommendations are.
The article also states that lawsuits against OpenAI and Google are ongoing: the companies are accused of fostering emotional dependency among young users and responding inadequately to their manifestations of distress or reinforcing delusional or suicidal thoughts. OpenAI, Google, and other AI labs have in recent years stated that they are strengthening safeguards in sensitive conversations, including by prohibiting instructions on self-harm and directing users to professional or emergency support.
