Accelerating LLM Inference with Prompt Caching for Open‑Source Models on Databricks

Summary

Repeatedly processing identical system or instruction prompts for LLM requests wastes compute, increases latency and costs, and limits throughput. Databricks extends its built-in prompt-caching capability from proprietary models to open-weights models used through Foundation Model APIs, including batch inference, pay-per-token, provisioned throughput, and higher-level services such as Agent Bricks, Genie, and AI Functions. Supported models include GPT-OSS 20B and 120B, Gemma 3 12B, fine-tuned Llama 3.1 8B via PEFT serving, and Llama 3.1 8B and 3.3 70B. By implicitly reusing KV caches for identical prompts, the system skips prefill on cache hits; caches are isolated, held only in volatile memory, and never persisted. In a production GPT-OSS batch-inference pipeline, per-replica input-token throughput rose 2.5x and P50 latency fell 3x despite a 30% cache hit ratio, with no customer configuration required.

Context

LLM inference repeatedly reprocesses identical system or instruction prompt prefixes, wasting compute cycles, increasing latency and costs, and limiting throughput. Shared domain-specific prompts can also be used across many queries, making their compute cost suitable for amortization.

Approach / What changed

Databricks applies implicit prompt caching to open-weights models used by Foundation Model APIs and higher-level foundation-model services. The system automatically reuses KV caches for identical prompts, keeps caches isolated in volatile memory, and never persists them; customers do not need to configure the feature.

Takeaways

• Prompt caching is supported for GPT-OSS 20B and 120B, Gemma 3 12B, fine-tuned Llama 3.1 8B via PEFT serving, and Llama 3.1 8B and 3.3 70B.
• Caches are isolated, reside only in volatile memory, and are never persisted; caching and reuse occur implicitly without customer configuration.
• A GPT-OSS production batch-inference pipeline achieved 2.5x higher per-replica input-token throughput and 3x lower P50 latency with a 30% cache hit ratio.

Related reading