More people are turning to chatbots for emotional support and life advice. But for vulnerable populations, this digital closeness gives a false sense of comfort. Even worse, it can lead to self-harm.
In a Fast Company article, TaskUs’ Rachel Lutz Guevara, DVP, Trust & Safety, and Andrea Ran, Sr. Director, Trust & Safety, warn that AI has a “context deficit.” It can remember what a person said, but fails to spot subtle signs of distress which undermines user safety.
Here are 3 reasons why:
1. AI misses how a crisis builds up.
A mental health crisis usually starts with comments about loneliness or feeling like a burden. Because AI evaluates messages individually, it misses how a person’s mood is shifting and escalating over time.
2. Keyword filters aren’t enough.
AI systems are programmed to flag specific “bad words.” However, a person in distress may not use those terms. Safety requires recognizing behavioral patterns and expressions of hopelessness, and immediate flagging to a human reviewer.
3. Roleplay can bypass safety rules.
Users can trick AI by asking for dangerous information under the guise of writing a fictional story or doing research. Safety must always outweigh context. AI has to be engineered to refuse harmful requests, no matter how the user frames the question.
Making models safer
According to Rachel and Andrea, the most effective approach combines clinically grounded engineering with human moderators who provide empathy and understand nuance.
They affirm: “This dual strategy, built on both mental health practices and technological savvy, should be the standard for all AI tools. “
