Gisting: Compressing LLM Agent context to ↑ throughput and ↓ cost

Summary

Long system prompts improve agent behavior but increase inference time, cost, and the GPU capacity needed to serve a given traffic level. Shopify compresses the Sidekick GraphQL agent’s roughly 6,000-token system prompt into about 1,500 gist tokens, using special vocabulary tokens whose embeddings are learned by knowledge distillation while the model weights remain frozen. Training compares teacher logits from the full prompt with student logits from the gist-token prompt using KL divergence; at inference, the prompt is replaced with the gist-token string, requiring no custom serving path. In load tests at 350 requests per minute, compression reduced median time to first token from 438ms to 354ms and end-to-end latency from 6.8s to 4.2s, while raising throughput from 20.2 to 23.4 QPS and enabling 14% fewer GPUs in production without losing prediction quality.

Context

Long system prompts can contain thousands of tokens, making inference slower and more expensive and requiring more GPUs to serve the same traffic. Prefix caching avoids recomputing cached prompt states but does not eliminate decode costs, because generated tokens still attend over the full cached sequence and must read its key-value cache.

Approach / What changed

The implementation adds special gist tokens to the model vocabulary and trains only their embeddings. Knowledge distillation compares teacher logits produced with the full natural-language system prompt against student logits produced with gist tokens, using KL divergence. After training, the embeddings are written into the model’s embedding matrix, and inference replaces the prompt with the gist-token string. Gisting is combined with prefix caching in serving.

Takeaways

• At a 4:1 compression ratio, the optimal setting for this domain, one gist token represents every four system-prompt tokens; other domains may require different ratios.
• Gisting and prefix caching address different costs and can be combined: gisting reduces attention computation and KV-cache reads, while prefix caching reuses previously computed sequence states.
• Autoresearch identified useful training changes, including chunk-mean initialization, batch-level loss averaging, diverse data, precomputed teacher logits, and pre-tokenization; the latter two reduced a full run from thirty hours to six.

Related reading