Loading post…
Together AIProblem & solution
How a global fintech scaled coding agent traffic with Dedicated Model Inference
Summary
A global fintech runs its coding assistant on GLM 5.2 through Together’s Dedicated Model Inference (DMI), but spiky engineering-hours traffic outpaced static capacity planning. DMI gives the customer’s teams direct control over endpoint provisioning, scaling, configuration, model changes, and performance data through APIs, UI, and CLI. The workload is peak-load and relatively low-TPS, with ISL p50/p90/p95 of roughly 81K/163K/178K tokens and RPS p50/p90/p95 of 3/6/7, so concurrency headroom mattered more than raw throughput; the deployment used dozens of B200s, multiple replicas, and 256K context. When a reshaped endpoint later reached near-100% prefill capacity, requests queued for one to three minutes and decode throughput fell to roughly 5 tokens per second, the team restored cache-session-aware routing and widened max-inflight-per-worker, pushing the change same-day with zero downtime.
Context
The customer’s coding-assistant traffic is concentrated in engineering hours, with sharp bursts in concurrency and prompt size as more teams adopt agents. Its requirements included self-service provisioning, programmatic observability, fast scaling, concurrency headroom, and the ability to change models and context configurations without renegotiation.
Approach / What changed
Dedicated Model Inference provided endpoint lifecycle control through an API, UI, and CLI, along with a metrics API for usage and performance data. The customer used those controls to reshape replicas, adjust cache-session-aware routing and max-inflight-per-worker settings without downtime, move from GLM 5.1 to GLM 5.2, and choose 256K/512K context instead of 1M to preserve concurrency headroom.
Takeaways
The workload’s ISL p50/p90/p95 was roughly 81K/163K/178K tokens, while RPS p50/p90/p95 was 3/6/7; concurrency headroom therefore mattered more than raw throughput.
A 192-second request was traced through metrics to a 2.3M-token pending-prefill backlog, showing that the delay was queuing rather than compute-bound.
The customer rejected a 1M-context configuration because doubling context would reduce the concurrency headroom needed by its peak-load, low-TPS workload, choosing 256K/512K instead.
FAQ
Why was concurrency prioritized over raw throughput for this coding workload?
Traffic was concentrated in engineering hours and arrived in sharp bursts, while request rates were relatively low. As more teams adopted coding agents, concurrency and prompt-size spikes mattered more than maximizing tokens per second.
How did the team fix the endpoint’s queuing incident?
The team restored a tuned cache-session-aware routing policy instead of DMI’s default cache-aware-by-hash policy and widened the max-inflight-per-worker threshold. The configuration change was applied the same day with zero downtime.
Why did the customer avoid a 1M-context configuration?
The team evaluated 1M context and turned it down because doubling context would cut the concurrency headroom required by its peak-load, low-TPS traffic. It chose to remain at 256K/512K context.
Source and freshness
- Source
- together.ai
- Published
- Added to Yomu
Summaries are generated from the original by a language model. Check the original before relying on it.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
The best engineering reads, once a week
One useful email. Unsubscribe anytime.