OpenAI announces plan for flagging mental health crises and adding teen controls

Over the past few weeks, a number of alarming stories have surfaced describing ChatGPT users who turned to the AI chatbot for help during mental health crises, only to have the tool encourage their plans for self-harm.

• In August, a Connecticut man killed his mother then himself after ChatGPT reportedly encouraged his paranoid delusions.

• In April, A 16-year-old died of suicide after conversations with ChatGPT in which the chatbot provided specific information about how to kill himself.

• In a New York Times op-ed, a mother shared the story of how her 29-year-old daughter had shared her suicidal thoughts with a ChatGPT “therapist” persona, yet it failed to alert anyone — something a real mental health professional would have done. ChatGPT helped the woman write her suicide note.

Today, ChatGPT maker OpenAI announced a 120-day plan to roll out new protections for identifying and helping users who may be facing a mental health crisis, including adding parental controls — even though they wouldn’t have helped in two of these cases.

The company plans to partner with councils of mental health and medical professionals to “shape a clear, evidence-based vision for how AI can support people’s well-being and help them thrive,” according to a blog post announcing the changes.

OpenAI said it also will route sensitive conversations to its “reasoning models,” which take longer to craft more accurate and helpful responses.

As kids head back to school, ChatGPT use is climbing as students widely adopt the tool for help with schoolwork. The planned parental control system will allow parents to link their teen’s account to theirs, control how ChatGPT responds, control features like chat history, and receive notifications when the system identifies “acute distress” in the teens’ conversations.