Measuring political bias in Claude

Summary

The post sets out an effort to make Claude politically even-handed: balanced, accurate, comprehensive, respectful, and reluctant to offer unsolicited political opinions or persuasion. It describes system-prompt guidance and reinforcement-learning-based character training, including traits that encourage neutral terminology, representation of competing perspectives, and restraint on sensitive political topics. For measurement, the Paired Prompts method compares responses to the same contentious issue framed from opposing ideological positions, scoring dimensions such as equal engagement and acknowledgment of opposing views; an automated version applies it to thousands of prompts across hundreds of stances. In the reported comparison, Claude Sonnet 4.5 was more even-handed than GPT-5 and Llama 4, and similar to Grok 4 and Gemini 2.5 Pro. The evaluation is open-sourced, with limitations including a focus on current US discourse and single-turn interactions.

Takeaways

1. 1

   The system prompt can substantially change Claude’s responses, but it is not foolproof and the model may still produce answers inconsistent with the intended principles.

2. 2

   Character training uses reinforcement learning against predefined traits, including avoiding rhetoric that could unduly alter political views or support political ads, propaganda, or ideological targeting.

3. 3

   Evaluation results may vary because each run generates fresh responses; the reported metrics average topic pairs equally and do not apply political-salience weighting.