Smart Routing in Unity AI Gateway: Match frontier quality with 30%+ lower cost per task

Summary

Unity AI Gateway’s Smart Routing, now in Beta, addresses the cost and choice overload of assigning coding tasks to increasingly diverse models and harnesses. It uses task-aware routing: a small, low-latency model classifies the initial task description and metadata, then a policy selects a cheaper, medium, or frontier model; Omnigent can also choose the coding harness and route sub-agents. The design keeps consecutive turns on one model to preserve cache-hit rates, while escalating complex work and delegating simpler tasks to lower-cost options. On internal coding workloads, Smart Routing achieved 35% savings and outperformed any single model at 65% of Opus 5’s cost; public benchmarks reported matched Opus 5 performance with 56% savings. The team identifies mid-session reassessment, real-session evaluation, and affordable switching as continuing challenges.

Context

Coding tasks vary widely in complexity, and users face choice overload when selecting among many models and harnesses. The post states that defaulting every task to the most capable model raises costs, while blunt spending caps or manual selection can limit productivity.

Approach / What changed

Smart Routing uses task-aware routing at the start of a session. A small, low-latency model extracts semantic fields from the task description and metadata, after which a single policy selects a cheaper, medium-sized, or frontier model. Omnigent extends the decision across coding harnesses and sub-agent launches.

Takeaways

• Task-aware routing keeps consecutive turns on the same model to preserve cache-hit rates, unlike per-request routing that can route each prompt independently.
• Omnigent can automatically select both a coding harness and model, and sub-agents can receive different routing decisions for planning, summarization, or parallel work.
• The team is investigating delayed routing, smaller sessions, and cheaper model switching because opening prompts are often underspecified and sessions can change subject over time.

Related reading