Reliable LLM Inference at Scale

Summary

Databricks describes an inference platform serving open-source and proprietary frontier models at more than 125T tokens per month, with reliability as the central challenge under spiky, multi-tenant demand. It identifies failures and latency risks from expensive GPU configurations, variable input and output lengths, long-context requests, multimodal preprocessing, and engine-level hangs. Its architecture combines rate limiting, capacity management, model units, Dicer load-aware sticky routing, and autoscaling; model units estimate multidimensional request cost and allocate capacity more predictably. For runtime reliability, prioritized black-box health checks trigger Kubernetes liveness restarts for silent hangs, completing detection and recovery in less than five minutes while eliminating false probe failures. Image-processing changes, including Torchvision processors and OMP_NUM_THREADS configuration, removed CPU throttling and increased completed requests per second by more than threefold with the same replicas and load; bursty workloads also achieved over 80% GPU savings versus static peak provisioning.

Context

LLM inference demand is growing exponentially and exhibits dramatic spikes, while multi-tenant serving must maintain both reliability and latency. Different workloads have different latency requirements, and failures can arise from GPU infrastructure, variable request costs, multimodal preprocessing, or inference-engine edge cases.

Approach / What changed

The platform uses rate limiting, capacity management, model units, Dicer load-aware routing with sticky sessions, and autoscaling. Runtime reliability relies on prioritized black-box health checks and Kubernetes liveness probes. Image-serving performance was improved by using Torchvision-based processors and configuring OMP_NUM_THREADS appropriately.

Takeaways

• Model units represent request cost using multidimensional estimates that account for input and output length, model and hardware characteristics, prefix caching, and multimodality.
• Dicer routes by server load measured in model units and uses sticky sessions, helping avoid hotspots, improve cache hit rates, and limit failure blast radius.
• Prioritized health checks reduced false liveness failures to zero; image-processing optimizations delivered more than 3x the completed requests per second with unchanged replicas and load.

Related reading