Prompting Claude's long context window

Summary

An evaluation of Claude’s 100,000-token context window tests prompting techniques for recalling specific information from long documents. Researchers built randomized collages from sections of a July 13 government document, using Claude-generated multiple-choice questions and filtering out questions it could not answer from the source passage alone. They compared a baseline with unrelated examples, dynamically selected contextual examples, and two or five examples, each with or without a scratchpad that instructed Claude Instant 1.2 to extract relevant quotes across roughly 70K and 95K token contexts. Scratchpads and contextual examples generally improved performance, with more examples performing better, although recall could decline when the answer passage was at the end. For Claude 2, the best prompting strategy raised performance from 0.939 to 0.961, a reported 36% reduction in errors; reproducible code was released in the Anthropic Cookbook.

Takeaways

1. 1

   A scratchpad that directs Claude to pull relevant quotes improved recall in head-to-head comparisons, with a small latency cost.

2. 2

   Examples drawn from other sections of the same context helped on both 70K and 95K documents, and five examples performed better than two.

3. 3

   Prompt additions could degrade recall when the relevant passage was at the end, because they increased the distance between that passage and the answer step.