Reliable LLM Inference at Scale

Summary

Databricks describes an inference platform serving open-source and proprietary frontier models at more than 125T tokens per month, with reliability as the central challenge under spiky, multi-tenant demand. It identifies failures and latency risks from expensive GPU configurations, variable input and output lengths, long-context requests, multimodal preprocessing, and engine-level hangs. Its architecture combines rate limiting, capacity management, model units, Dicer load-aware sticky routing, and autoscaling; model units estimate multidimensional request cost and allocate capacity more predictably. For runtime reliability, prioritized black-box health checks trigger Kubernetes liveness restarts for silent hangs, completing detection and recovery in less than five minutes while eliminating false probe failures. Image-processing changes, including Torchvision processors and OMP_NUM_THREADS configuration, removed CPU throttling and increased completed requests per second by more than threefold with the same replicas and load; bursty workloads also achieved over 80% GPU savings versus static peak provisioning.

Takeaways

1. 1

   Model units represent request cost using multidimensional estimates that account for input and output length, model and hardware characteristics, prefix caching, and multimodality.

2. 2

   Dicer routes by server load measured in model units and uses sticky sessions, helping avoid hotspots, improve cache hit rates, and limit failure blast radius.

3. 3

   Prioritized health checks reduced false liveness failures to zero; image-processing optimizations delivered more than 3x the completed requests per second with unchanged replicas and load.