---
title: "How a global fintech scaled coding agent traffic with Dedicated Model Inference"
description: "A global fintech runs its coding assistant on GLM 5.2 through Together’s Dedicated Model Inference (DMI), but spiky engineering-hours traffic outpaced static capacity planning. DMI gives the customer’s teams direct control over endpoint provisioning, scaling, configuration, model changes, and performance data through APIs, UI, and CLI. The workload is peak-load and relatively low-TPS, with ISL p50/p90/p95 of roughly 81K/163K/178K tokens and RPS p50/p90/p95 of 3/6/7, so concurrency headroom mattered more than raw throughput; the deployment used dozens of B200s, multiple replicas, and 256K context. When a reshaped endpoint later reached near-100% prefill capacity, requests queued for one to three minutes and decode throughput fell to roughly 5 tokens per second, the team restored cache-session-aware routing and widened max-inflight-per-worker, pushing the change same-day with zero downtime."
---

# How a global fintech scaled coding agent traffic with Dedicated Model Inference

[Together AI](https://yomu.fyi/company/together-ai) · Sep 18, 2026

**Type:** Problem & solution

## Summary

A global fintech runs its coding assistant on GLM 5.2 through Together’s Dedicated Model Inference (DMI), but spiky engineering-hours traffic outpaced static capacity planning. DMI gives the customer’s teams direct control over endpoint provisioning, scaling, configuration, model changes, and performance data through APIs, UI, and CLI. The workload is peak-load and relatively low-TPS, with ISL p50/p90/p95 of roughly 81K/163K/178K tokens and RPS p50/p90/p95 of 3/6/7, so concurrency headroom mattered more than raw throughput; the deployment used dozens of B200s, multiple replicas, and 256K context. When a reshaped endpoint later reached near-100% prefill capacity, requests queued for one to three minutes and decode throughput fell to roughly 5 tokens per second, the team restored cache-session-aware routing and widened max-inflight-per-worker, pushing the change same-day with zero downtime.

## Context

The customer’s coding-assistant traffic is concentrated in engineering hours, with sharp bursts in concurrency and prompt size as more teams adopt agents. Its requirements included self-service provisioning, programmatic observability, fast scaling, concurrency headroom, and the ability to change models and context configurations without renegotiation.

## Approach / What changed

Dedicated Model Inference provided endpoint lifecycle control through an API, UI, and CLI, along with a metrics API for usage and performance data. The customer used those controls to reshape replicas, adjust cache-session-aware routing and max-inflight-per-worker settings without downtime, move from GLM 5.1 to GLM 5.2, and choose 256K/512K context instead of 1M to preserve concurrency headroom.

## Takeaways

- The workload’s ISL p50/p90/p95 was roughly 81K/163K/178K tokens, while RPS p50/p90/p95 was 3/6/7; concurrency headroom therefore mattered more than raw throughput.
- A 192-second request was traced through metrics to a 2.3M-token pending-prefill backlog, showing that the delay was queuing rather than compute-bound.
- The customer rejected a 1M-context configuration because doubling context would reduce the concurrency headroom needed by its peak-load, low-TPS workload, choosing 256K/512K instead.

## FAQ

### Why was concurrency prioritized over raw throughput for this coding workload?

Traffic was concentrated in engineering hours and arrived in sharp bursts, while request rates were relatively low. As more teams adopted coding agents, concurrency and prompt-size spikes mattered more than maximizing tokens per second.

### How did the team fix the endpoint’s queuing incident?

The team restored a tuned cache-session-aware routing policy instead of DMI’s default cache-aware-by-hash policy and widened the max-inflight-per-worker threshold. The configuration change was applied the same day with zero downtime.

### Why did the customer avoid a 1M-context configuration?

The team evaluated 1M context and turned it down because doubling context would cut the concurrency headroom required by its peak-load, low-TPS traffic. It chose to remain at 256K/512K context.

**Tags:** [GPU](https://yomu.fyi/topic/gpu), [LLMs](https://yomu.fyi/topic/llm), [Model Serving](https://yomu.fyi/topic/model-serving), [Observability](https://yomu.fyi/topic/observability), [Scalability](https://yomu.fyi/topic/scalability)

- Source: [Together AI](https://www.together.ai/blog/global-fintech-scales-coding-agent-traffic-with-dedicated-model-inference)
- Source URL: https://www.together.ai/blog/global-fintech-scales-coding-agent-traffic-with-dedicated-model-inference
- Ingested by Yomu: 2026-09-19T10:13:48.947Z

[Read original post](https://www.together.ai/blog/global-fintech-scales-coding-agent-traffic-with-dedicated-model-inference)