---
title: "How to serve trillions of tokens for trillion-parameter coding agents"
description: "Serving inference for coding agents requires trillion-parameter sequence models like Moonshot AI’s Kimi K2.6 to process trillions of input and output tokens at extremely high relative and absolute performance. To be economically viable, these services must achieve large fractions of hardware peak performance, using contemporary matrix math accelerators like Tensor Cores at petaFLOP per second scale. Modal optimized by batching multiple sequences together, caching KV state, and patching SGLang with NVFP4 micro-scaling on Blackwell GPUs like the B200. Per-replica performance improved 2.8x per user and 5.6x across users, transforming ruinously expensive into a price-competitive service. One deployment processed hundreds of billions of tokens daily and trillions total, showing that careful engineering and benchmarking are key to scaling inference."
---

# How to serve trillions of tokens for trillion-parameter coding agents

[Modal](https://yomu.fyi/company/modal) · Sep 23, 2026

**Type:** Explainer

## Summary

Serving inference for coding agents requires trillion-parameter sequence models like Moonshot AI’s Kimi K2.6 to process trillions of input and output tokens at extremely high relative and absolute performance. To be economically viable, these services must achieve large fractions of hardware peak performance, using contemporary matrix math accelerators like Tensor Cores at petaFLOP per second scale. Modal optimized by batching multiple sequences together, caching KV state, and patching SGLang with NVFP4 micro-scaling on Blackwell GPUs like the B200. Per-replica performance improved 2.8x per user and 5.6x across users, transforming ruinously expensive into a price-competitive service. One deployment processed hundreds of billions of tokens daily and trillions total, showing that careful engineering and benchmarking are key to scaling inference.

## Approach / What changed

Optimized inference by deploying Kimi K2.6 on Blackwell GPUs with NVFP4 micro-scaling, using batching and KV cache management, patching SGLang for improved decode rates, and scaling single-container replicas across deployments.

## Takeaways

- Batching multiple sequences together during forward passes reduces per-token computation cost by amortizing work across requests, enabling higher throughput per GPU.
- Caching the internal state (KV cache) from prior forward passes avoids recomputation, significantly improving decode token generation rates in autoregressive models.
- Performance metrics like tokens per minute per GPU and speculative accept length are data-dependent, varying with output length and code versus prose content.

## FAQ

### Why is batching multiple requests necessary for inference?

Batching multiple sequences together amortizes the expensive per-token forward pass computations across multiple requests. This allows each sequence to benefit from the same GPU work, dramatically increasing throughput per GPU and making the service economically viable.

### What is the key-value cache and how does it improve speed?

The KV cache stores internal state from previous forward passes, avoiding recomputation during decode steps. This dramatically improves the rate at which output tokens are generated after the initial prefill and reduces the compute per step.

### How does data variation affect benchmark results?

Performance metrics like tokens-per-minute per GPU depend on output length, and speculative accept lengths vary by content (code vs prose). Benchmarking must use data matching production to produce reliable results; changing data can change results without system changes.

**Tags:** [AI Agents](https://yomu.fyi/topic/ai-agents), [Blackwell](https://yomu.fyi/topic/blackwell), [GPU](https://yomu.fyi/topic/gpu), [Inference](https://yomu.fyi/topic/inference), [SGLang](https://yomu.fyi/topic/sglang)

- Source: [Modal](https://modal.com/blog/trillion-tokens-trillion-parameters)
- Source URL: https://modal.com/blog/trillion-tokens-trillion-parameters
- Ingested by Yomu: 2026-09-23T21:00:11.275Z

[Read original post](https://modal.com/blog/trillion-tokens-trillion-parameters)