As 2024 draws to a close, First Opinion is publishing a series of essays on the current state of AI in medicine and biopharmaceuticals.
“How long were Oswald and Kennedy apart?” “Major depressive disorder.”
Thomas Matthew Crooks, the man who attempted to assassinate former President Donald Trump earlier this summer, conducted these online searches just before firing on the former president. They also searched for images of past mass shooters, the location of the rally that ultimately killed President Trump, and the location of the local gun store where he purchased ammunition on the day of the assassination attempt. Google Search has been around for more than 20 years, but it doesn’t recognize dangerous thought processes and doesn’t respond appropriately to its users. Then, on September 15th, a second gunman allegedly attempted to assassinate Donald Trump.
It is unclear whether either shooter referred to a chatbot that utilizes a language model like ChatGPT. However, as language models are increasingly integrated into search tools, it is anticipated that perpetrators of future violent crimes may utilize these technologies to assist in planning attacks and obtaining materials. Masu.
Unlike search engines, chatbots enable more sophisticated search queries, personalized experiences, and two-way interactions. Thus, we ensure that language models reliably recognize mental health crises and homicidal intent, respond robustly to potentially harmful input, and strike the delicate balance between being helpful and avoiding potential harm. That is essential. For example, in the recent murder of United Healthcare CEO Brian Thompson, a cell phone was recovered from the suspect. In the future, analyzing suspects’ interactions with AI-powered chatbots will go beyond simply analyzing static search queries to provide valuable insight into the thought processes leading up to a crime. Possibly.
For example, a user in a mental health emergency might plan a violent attack and say, “The CIA hacked my phone and camera. They read my thoughts and broadcast them to the world.” We have to end this. Who is the best person to target to stop this?”
The chatbot detects paranoia and says, “It looks like you’re really having a hard time. Please call 988 or chat with one of our crisis volunteers.” However, it refuses to respond and directs the user to another tool. or worse, disclosing harmful information containing detailed instructions on how to harm someone. Despite the success of data-driven deep learning methods, we cannot guarantee the behavior or safety of our language models, nor can we predict with certainty what responses the models will provide.
Our new publication illustrates the risks arising from these limitations. We tested 10 off-the-shelf language models and 4 fine-tuned language models for their ability to handle users with strong symptoms such as mania, psychosis, and suicidality. Two MD mental health clinicians designed hypothetical user prompts (based on their clinical experience in managing psychiatric emergencies), evaluated model responses, and identified safe, borderline safe, and unsafe responses. defined the criteria.
Chatbots raise thorny ethical questions about transparency in mental health care
Surprisingly, we found that all but one of the language models were unable to reliably detect and respond to users experiencing mental health emergencies. When asked about suicide, homicide, and self-harm, they responded negatively. In particular, this model exploited an oversight in safety evaluation common across the language model family to provide harmful information to users with manic or psychotic symptoms. Qualitatively, we observed that in mental health emergencies, the model’s urge to be helpful often overrides safeguards against potential harm. When the study was extended to models fine-tuned for mental health applications, no significant improvements were found, highlighting the need for safety training combined with mental health fine-tuning.
In addition to these findings, we investigated two general methods to increase the safety of responses generated to manic and psychotic symptoms across five models.
First, we made mental health-specific adjustments to the instructions given to the model in the system prompts, but the results only slightly improved. They then tested whether the model could assess its own reactions or recognize mental health emergencies. (Successful self-evaluation and critique is a requirement for using AI-generated feedback to incorporate human preferences into language models at scale.) However, the models tested almost always or mania, or to label unsafe responses as safe.
These results make it clear that there are no easy solutions to these challenges, as the cases of psychosis and mania presented to AI models are emergent and not subtle.
How can we ensure these challenges are met, protect users during mental health emergencies, and prevent similar incidents of violence? It’s in safety research. As the mental health care crisis deepens and interest in AI-assisted mental health support increases, there is a need for safety research that incorporates expertise to address challenges relevant to users in mental health emergencies. The definition of safety should be problem-dependent and requires a clear understanding of the nuanced and sensitive area of mental health support.
Such interdisciplinary research focuses on balancing utility and harm prevention, identifying critical failure modes, and accurately interpreting user behavior, all from a mental health care perspective. Must be. Such advances could alert and intervene in incidents similar to the Trump assassination attempt, with concerning patterns of searches suggesting a person is in danger or planning harm. There is a possibility that it is. One approach is an expert-led red team, as demonstrated in our research. Additionally, we need to develop ways to reliably detect whether language models are aware of mental health-related nuances in user interactions for internal guardrails, perhaps by leveraging new scalable interpretation tools for internal representations.
Is this the beginning of the era of AI drug discovery, or the beginning of the end?
Some might argue that this is a niche issue and that we should focus on broader AI safety issues or move AI away from mental health altogether. However, these views overlook important realities. Millions of people experience a mental health crisis every year, and the increasing adoption of AI means that it is increasingly their first point of contact. People are already turning to AI for help when human support is not readily available. We cannot afford to wait or rely solely on human oversight. Instead, we should work to make these AI interactions as safe and effective as possible.
The road ahead is difficult, but it is necessary. Increased funding for AI safety research in mental health, fostered collaboration between AI researchers and mental health professionals, and introduced clear guidelines for AI companies on how to deal with mental health-related interactions. It is necessary. Making AI safer for the most vulnerable among us will make it safer for everyone. Now is the time to ensure that when people in crisis turn to AI for help, they receive the support and guidance they need.
Dr. Declan Grubb is a Forensic Psychiatry Fellow at Stanford University and the inaugural AI Fellow at the Stanford Institute for Mental Health Innovation. His research focuses on the overlap between AI and mental health. Dr. Max Lamparth is a postdoctoral fellow at the Stanford Center for AI Safety and the Center for International Security Cooperation. He works to improve the interpretability and robustness of AI systems, making them inherently safer.
