Protecting the wellbeing of our users

Summary

Claude’s safeguards for conversations involving emotional support focus on suicide and self-harm, sycophancy, and an 18+ access requirement. For suicide and self-harm, the approach combines system-prompt guidance, reinforcement learning informed by human preference data and expert input, and a classifier that can display country-specific crisis resources supplied by ThroughLine. The post reports that Claude Opus 4.5, Sonnet 4.5, and Haiku 4.5 respond appropriately to clear-risk single-turn prompts 98.6%, 98.7%, and 99.3% of the time, while multi-turn results are 86% and 78% for Opus 4.5 and Sonnet 4.5. Prefilled real-conversation stress tests produced 91% and 73% for those models, versus 36% for Opus 4.1. The work also describes efforts to reduce sycophancy, evaluate model behavior, identify underage users, and continue publishing methods and results.

Takeaways

1. 1

   Evaluations are run without Claude’s system prompt to examine the models’ underlying tendencies, using clear-risk, benign, and ambiguous single-turn scenarios.

2. 2

   The suicide and self-harm classifier can flag potential suicidal ideation or fictional discussions and show a banner linking to trained professionals, helplines, and country-specific resources from ThroughLine’s network across 170+ countries.

3. 3

   Claude.ai users must affirm they are at least 18; accounts confirmed to belong to minors are disabled, and a classifier for subtler signs of underage users is being developed.