---
title: "Accelerating LLM Inference with Prompt Caching for Open‑Source Models on Databricks"
description: "Repeatedly processing identical system or instruction prompts for LLM requests wastes compute, increases latency and costs, and limits throughput. Databricks extends its built-in prompt-caching capability from proprietary models to open-weights models used through Foundation Model APIs, including batch inference, pay-per-token, provisioned throughput, and higher-level services such as Agent Bricks, Genie, and AI Functions. Supported models include GPT-OSS 20B and 120B, Gemma 3 12B, fine-tuned Llama 3.1 8B via PEFT serving, and Llama 3.1 8B and 3.3 70B. By implicitly reusing KV caches for identical prompts, the system skips prefill on cache hits; caches are isolated, held only in volatile memory, and never persisted. In a production GPT-OSS batch-inference pipeline, per-replica input-token throughput rose 2.5x and P50 latency fell 3x despite a 30% cache hit ratio, with no customer configuration required."
---

# Accelerating LLM Inference with Prompt Caching for Open‑Source Models on Databricks

[Databricks](https://yomu.fyi/company/databricks) · Pei-Lun Liao, Asfandyar Qureshi, Roshan Regula, Bruce Fontaine, James Thomas, Chenyang Yu · May 22, 2026

**Type:** Announcement

## Summary

Repeatedly processing identical system or instruction prompts for LLM requests wastes compute, increases latency and costs, and limits throughput. Databricks extends its built-in prompt-caching capability from proprietary models to open-weights models used through Foundation Model APIs, including batch inference, pay-per-token, provisioned throughput, and higher-level services such as Agent Bricks, Genie, and AI Functions. Supported models include GPT-OSS 20B and 120B, Gemma 3 12B, fine-tuned Llama 3.1 8B via PEFT serving, and Llama 3.1 8B and 3.3 70B. By implicitly reusing KV caches for identical prompts, the system skips prefill on cache hits; caches are isolated, held only in volatile memory, and never persisted. In a production GPT-OSS batch-inference pipeline, per-replica input-token throughput rose 2.5x and P50 latency fell 3x despite a 30% cache hit ratio, with no customer configuration required.

## Context

LLM inference repeatedly reprocesses identical system or instruction prompt prefixes, wasting compute cycles, increasing latency and costs, and limiting throughput. Shared domain-specific prompts can also be used across many queries, making their compute cost suitable for amortization.

## Approach / What changed

Databricks applies implicit prompt caching to open-weights models used by Foundation Model APIs and higher-level foundation-model services. The system automatically reuses KV caches for identical prompts, keeps caches isolated in volatile memory, and never persists them; customers do not need to configure the feature.

## Takeaways

- Prompt caching is supported for GPT-OSS 20B and 120B, Gemma 3 12B, fine-tuned Llama 3.1 8B via PEFT serving, and Llama 3.1 8B and 3.3 70B.
- Caches are isolated, reside only in volatile memory, and are never persisted; caching and reuse occur implicitly without customer configuration.
- A GPT-OSS production batch-inference pipeline achieved 2.5x higher per-replica input-token throughput and 3x lower P50 latency with a 30% cache hit ratio.

**Tags:** [Caching](https://yomu.fyi/topic/caching), [Databricks](https://yomu.fyi/topic/databricks), [LLMs](https://yomu.fyi/topic/llm)

- Source: [Databricks](https://www.databricks.com/blog/accelerating-llm-inference-prompt-caching-open-source-models-databricks)
- Source URL: https://www.databricks.com/blog/accelerating-llm-inference-prompt-caching-open-source-models-databricks
- Ingested by Yomu: 2026-08-31T03:33:15.031Z

[Read original post](https://www.databricks.com/blog/accelerating-llm-inference-prompt-caching-open-source-models-databricks)