---
title: "Model Routing Is Simple. Until It Isn’t."
description: "Model routing in agentic systems is presented as a systems optimization problem rather than a simple classification task. On 417 AppWorld Test Challenge tasks with the same CodeAct agent, Claude Sonnet 4.6 cost $79 total ($0.19 per task), versus $155 ($0.37 per task) for GPT-4.1, despite GPT-4.1 having lower listed token prices; cache-read pricing and repeated context changed the effective result. The discussion also shows why task difficulty and model speed are incomplete signals: hidden execution work, specialization, governance constraints, routing overhead, cache state, hardware, and endpoint load all affect decisions. The described router optimizes cost, quality, and latency together, producing configurable cost-accuracy operating points; one latency-oriented configuration reached 84% accuracy for $93 and 83 seconds, while adding about 6 ms and 2 kB per task."
---

# Model Routing Is Simple. Until It Isn’t.

[Hugging Face](https://yomu.fyi/company/hugging-face) · Yara Rizk, Eyal Shnarch, Jason Tsay, Merve Unuvar · Jul 15, 2026

**Type:** Explainer

## Summary

Model routing in agentic systems is presented as a systems optimization problem rather than a simple classification task. On 417 AppWorld Test Challenge tasks with the same CodeAct agent, Claude Sonnet 4.6 cost $79 total ($0.19 per task), versus $155 ($0.37 per task) for GPT-4.1, despite GPT-4.1 having lower listed token prices; cache-read pricing and repeated context changed the effective result. The discussion also shows why task difficulty and model speed are incomplete signals: hidden execution work, specialization, governance constraints, routing overhead, cache state, hardware, and endpoint load all affect decisions. The described router optimizes cost, quality, and latency together, producing configurable cost-accuracy operating points; one latency-oriented configuration reached 84% accuracy for $93 and 83 seconds, while adding about 6 ms and 2 kB per task.

## Context

Model selection based only on listed model prices, estimated task difficulty, or model speed does not capture agent workload behavior. Effective cost is affected by caching and repeated context; task complexity may emerge during execution; and production routing must also account for quality, latency, specialization, compliance, privacy, data residency, approved models, infrastructure state, and routing overhead.

## Approach / What changed

The router treats model selection as a lightweight optimization problem over cost, quality, and latency rather than as classification. It supports multiple operating points along a cost-accuracy frontier, allowing configurations to prioritize latency, cost, or accuracy while accounting for execution and serving conditions. The optimization adds roughly 6 ms and 2 kB of memory per task.

## Takeaways

- Across 417 AppWorld Test Challenge tasks using the same CodeAct agent, Claude Sonnet 4.6 cost $79 total, while GPT-4.1 cost $155, because Sonnet benefited more from lower cache-read pricing despite higher listed token prices and longer trajectories.
- Task difficulty is often unknown at routing time because apparently simple requests can trigger retrieval, compliance checks, tool use, and refinement, while technical prompts may be handled efficiently by smaller specialized models.
- A latency-optimized router configuration achieved 84% accuracy for $93 and 83 seconds, reducing cost by 21% and latency by 9% versus running Opus alone, with a 4% accuracy drop.

**Tags:** [Architecture](https://yomu.fyi/topic/architecture), [LLMs](https://yomu.fyi/topic/llm), [Performance](https://yomu.fyi/topic/performance)

- Source: [Hugging Face](https://huggingface.co/blog/ibm-research/model-routing-is-simple-until-it-isnt)
- Source URL: https://huggingface.co/blog/ibm-research/model-routing-is-simple-until-it-isnt
- Ingested by Yomu: 2026-08-29T04:01:02.610Z

[Read original post](https://huggingface.co/blog/ibm-research/model-routing-is-simple-until-it-isnt)