Claude 2

Summary

Claude 2 is introduced as a new model with improved performance, longer responses, expanded coding, math, and reasoning capabilities, and access through both an API and the claude.ai beta website. It supports prompts of up to 100K tokens and can produce documents of a few thousand tokens, enabling work across hundreds of pages of technical documentation or a book. Reported results include 76.5% on the Bar exam multiple-choice section versus 73.0% for Claude 1.3, 71.2% on Codex HumanEval versus 56.0%, and 88.0% on GSM8k versus 85.2%. Anthropic says internal red-team evaluation found Claude 2 2x better at giving harmless responses than Claude 1.3, while acknowledging that jailbreaks remain possible and inappropriate outputs can occur. The API is offered to businesses at the same price as Claude 1.3, and the chat experience is available in the US and UK, with broader availability planned.

Takeaways

1. 1

   Jasper reports that Claude 2 is particularly strong for long-form, low-latency use cases and offers a 3X larger context window for remixing existing content.

2. 2

   Sourcegraph’s Cody uses Claude 2 to pass up to 100K tokens of codebase context and draw on training data covering newer frameworks and libraries.

3. 3

   The chat experience is an open beta; users are warned that Claude can generate inappropriate responses and should not use it for physical or mental health and well-being.