---
title: "Reliable LLM Inference at Scale"
description: "Databricks describes an inference platform serving open-source and proprietary frontier models at more than 125T tokens per month, with reliability as the central challenge under spiky, multi-tenant demand. It identifies failures and latency risks from expensive GPU configurations, variable input and output lengths, long-context requests, multimodal preprocessing, and engine-level hangs. Its architecture combines rate limiting, capacity management, model units, Dicer load-aware sticky routing, and autoscaling; model units estimate multidimensional request cost and allocate capacity more predictably. For runtime reliability, prioritized black-box health checks trigger Kubernetes liveness restarts for silent hangs, completing detection and recovery in less than five minutes while eliminating false probe failures. Image-processing changes, including Torchvision processors and OMP_NUM_THREADS configuration, removed CPU throttling and increased completed requests per second by more than threefold with the same replicas and load; bursty workloads also achieved over 80% GPU savings versus static peak provisioning."
---

# Reliable LLM Inference at Scale

[Databricks](https://yomu.fyi/company/databricks) · Ying Chen, Wendy Hu, Ankit Mathur, Mike Eastham, Pei-Lun Liao, Wai Wu, Arjun DCunha · May 27, 2026

**Type:** Explainer

## Summary

Databricks describes an inference platform serving open-source and proprietary frontier models at more than 125T tokens per month, with reliability as the central challenge under spiky, multi-tenant demand. It identifies failures and latency risks from expensive GPU configurations, variable input and output lengths, long-context requests, multimodal preprocessing, and engine-level hangs. Its architecture combines rate limiting, capacity management, model units, Dicer load-aware sticky routing, and autoscaling; model units estimate multidimensional request cost and allocate capacity more predictably. For runtime reliability, prioritized black-box health checks trigger Kubernetes liveness restarts for silent hangs, completing detection and recovery in less than five minutes while eliminating false probe failures. Image-processing changes, including Torchvision processors and OMP\_NUM\_THREADS configuration, removed CPU throttling and increased completed requests per second by more than threefold with the same replicas and load; bursty workloads also achieved over 80% GPU savings versus static peak provisioning.

## Context

LLM inference demand is growing exponentially and exhibits dramatic spikes, while multi-tenant serving must maintain both reliability and latency. Different workloads have different latency requirements, and failures can arise from GPU infrastructure, variable request costs, multimodal preprocessing, or inference-engine edge cases.

## Approach / What changed

The platform uses rate limiting, capacity management, model units, Dicer load-aware routing with sticky sessions, and autoscaling. Runtime reliability relies on prioritized black-box health checks and Kubernetes liveness probes. Image-serving performance was improved by using Torchvision-based processors and configuring OMP\_NUM\_THREADS appropriately.

## Takeaways

- Model units represent request cost using multidimensional estimates that account for input and output length, model and hardware characteristics, prefix caching, and multimodality.
- Dicer routes by server load measured in model units and uses sticky sessions, helping avoid hotspots, improve cache hit rates, and limit failure blast radius.
- Prioritized health checks reduced false liveness failures to zero; image-processing optimizations delivered more than 3x the completed requests per second with unchanged replicas and load.

**Tags:** [Autoscaling](https://yomu.fyi/topic/autoscaling), [GPU](https://yomu.fyi/topic/gpu), [Kubernetes](https://yomu.fyi/topic/kubernetes), [LLMs](https://yomu.fyi/topic/llm), [Model Serving](https://yomu.fyi/topic/model-serving)

- Source: [Databricks](https://www.databricks.com/blog/reliable-llm-inference-scale)
- Source URL: https://www.databricks.com/blog/reliable-llm-inference-scale
- Ingested by Yomu: 2026-08-31T03:32:50.568Z

[Read original post](https://www.databricks.com/blog/reliable-llm-inference-scale)