---
title: "Online Learning for Cost-Efficient LLM Routing"
description: "Ramp describes an internal LLM gateway that processes trillions of tokens daily and uses dynamic failure-aware routing to reduce AI spend. It combines an EWMA of genuine provider-side failure rates with Thompson Sampling over a Normal-Inverse-Gamma posterior for the mean and variance of log-latency. For each request, the router estimates the probability of outright failure or missing the caller's deadline, incorporates model cost, and orders options while preserving caller-defined model preferences and fallbacks. In a reranker with a six-second deadline, the strategy reacted to Gemini 3.1 Flash Lite rate limiting by switching to GPT 5.4 Nano Flex, whose latency matched Standard at half the price. Larger experiments reported more than 25% cost savings without degradation, while a streaming-equivalent deployment for Ramp Inspect produced 30% savings."
---

# Online Learning for Cost-Efficient LLM Routing

[Ramp](https://yomu.fyi/company/ramp) · Kedar Thakkar · Jul 20, 2026

**Type:** Problem & solution

## Summary

Ramp describes an internal LLM gateway that processes trillions of tokens daily and uses dynamic failure-aware routing to reduce AI spend. It combines an EWMA of genuine provider-side failure rates with Thompson Sampling over a Normal-Inverse-Gamma posterior for the mean and variance of log-latency. For each request, the router estimates the probability of outright failure or missing the caller's deadline, incorporates model cost, and orders options while preserving caller-defined model preferences and fallbacks. In a reranker with a six-second deadline, the strategy reacted to Gemini 3.1 Flash Lite rate limiting by switching to GPT 5.4 Nano Flex, whose latency matched Standard at half the price. Larger experiments reported more than 25% cost savings without degradation, while a streaming-equivalent deployment for Ramp Inspect produced 30% savings.

## Context

Ramp needed to improve reliability and efficiency across a centralized LLM gateway while accounting for changing latency, provider failures, caller-specific deadlines, model preferences, and relative service costs. Latency patterns across OpenAI flex and default tiers were not sufficiently predictable for a fixed routing strategy.

## Approach / What changed

The gateway tracks genuine provider-side failures with an exponentially weighted moving average and models log-latency using Thompson Sampling with a conjugate Normal-Inverse-Gamma prior. It estimates each option's probability of failure or deadline violation, combines that risk with relative cost, then sorts the caller's preferred models and tiers for routing with fallback support.

## Takeaways

- The routing objective treats a request as a bad outcome when it fails outright or exceeds its deadline after avoiding failure, making latency use-case specific.
- Pooled sufficient statistics in Redis support online updates to the Normal-Inverse-Gamma latency posterior for each context slice.
- The strategy delivered over 25% savings in a larger experiment and 30% savings for Ramp Inspect's streaming-equivalent deployment.

**Tags:** [LLMs](https://yomu.fyi/topic/llm), [Model Routing](https://yomu.fyi/topic/model-routing), [Redis](https://yomu.fyi/topic/redis)

- Source: [Ramp](https://builders.ramp.com/post/thompson-sampling-model-routing)
- Source URL: https://builders.ramp.com/post/thompson-sampling-model-routing
- Ingested by Yomu: 2026-09-01T01:35:19.606Z

[Read original post](https://builders.ramp.com/post/thompson-sampling-model-routing)