Loading…
Topic
27 posts about Model Routing. Every summary links to the original.
See the idea
Antigravity SDK's hybrid architecture separates planning from execution: a cloud model plans, local models run most tokens on-device. See how routing, execution, and the data boundary fit together.
A benchmark can support two different verdicts: pass@1 on the first attempt, or pass@k with retries measured as cost per solved task. This comparison shows which reading matches your workload.
Understand how to choose between frontier models, efficient alternatives, and routing systems by weighing capability gains against token costs and latency constraints.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Level 5 covers operability for LLM systems: daily running, cost control, provider outage survival, and fast shutdown on misbehavior. A single gateway funnels all model traffic, measuring cost, routing requests, enabling failover, and hosting an instantaneous kill switch; a shared decision_id threads metrics, logs, traces, and the audit ledger. Observability supplements RED metrics with a decision layer capturing auto vs. human splits, guardrail blocks, judge disagreements, token spend per tenant and capability, and shadow-eval pass ratios via tiered structured logs. Cost controls attribute spending at the gateway per tenant and capability, with hard budget ceilings; optimization follows order: skip model, batch, cache, right-size, trim—each step profiled against quality. The platform spine includes stateless agent services, the gateway, an audit datastore, cache, secrets store, and object storage—scoped with least-privilege IAM and encryption, avoiding queues or vector stores until warranted.
Together Link connects coding agents to over 40 production-tier open models on Together AI, cutting model spend by over 50% compared to premium closed models. It integrates with Claude Code, Claude Desktop, Codex, OpenCode, and Pi without requiring any workflow changes. Open models such as Kimi K3, GLM 5.3, DeepSeek V4.1 Flash, and GLM 5.3 Flash now handle both complex architectural tasks and everyday fixes at a fraction of the per-token cost. A per-session router reads each session’s first task and routes it to either a fast, low-cost model or a frontier model; when no Anthropic key is provided it routes between GLM 5.3 and GLM 5.3 Flash, and Together AI serves the largest share of OpenRouter tokens for these models. Billing uses the existing Together API key with pay-as-you-go or credit packs, and a per-session tracker shows what the session would have cost on Opus 5.5.
Cloudflare released Auto Router in public beta through AI Gateway. Set the model to “cloudflare/auto” and the gateway automatically routes each request to a model capable enough for the task, filtering by credentials, access controls, and provider health. A multi-head classifier on Workers AI assigns probabilities across 14 task categories and rates complexity, ambiguity, stakes, and context dependence on a 1-5 scale. A separate scoring matrix combines those signals with model benchmark results and token prices, applying an adaptive cost penalty and switching costs for cached contexts in long sessions. On an internal general knowledge-work benchmark of 97 tasks with three samples each, Auto Router achieved 86.6% success at $0.0084 per success, costing 35% of Claude Opus 5.5 and 80% of GPT-6 Sol while remaining free during beta.
Cloudflare AI Gateway’s User Insights now provides context beyond model names and request counts, using a dedicated Worker to classify conversations by task type such as coding, research, summarization, or formatting. The “model overkill” view identifies conversations where a high-capability reasoning model is used for simple tasks, showing which users or agents are driving that usage and the associated cost. Teams can analyze conversation turns to see how many back-and-forth exchanges a task requires, helping uncover inefficiencies in prompts or workflows. The Potential Savings view and Auto Router (closed beta) use task and model-fit signals to automatically route requests to appropriate models without a separate rule for each workload. These features are free for AI Gateway users and require stable user_id and session_id for analysis, with optional Access integration.
The post presents an open model AI stack for developers moving from closed-source models, focusing on ownership, control, and economics. It organizes the stack into independent layers: model, inference, gateways and routers, harness, and tools such as Skills and MCP. Models generate responses from prompts; large models generally handle ambiguity, multi-step reasoning, and long context better, while small models can be faster and cheaper for tightly specified work. The separation lets developers choose providers and models independently and experiment with newly released models without rebuilding their workflow, while the post uses Kimi K3 and GLM 5.3 Flash to illustrate differences in active parameters, capability, speed, and cost.
Bring your own LLM (BYO LLM) is an architecture in which teams choose the model powering their AI agents while a platform supplies communication infrastructure. The division leaves model selection, prompts, reasoning logic, tool definitions, and called systems with the team, while channels, conversation context, audio streaming, identity resolution, and human escalation remain with the platform. The post explains why bundled models create quality, migration, compliance, and cost-control constraints, including the inability to route simple interactions to smaller models and complex work to frontier models. It describes adding a third-party model to an existing contact center through middleware, a shared message handler, tools and knowledge, channel registration, and context-preserving handoff. Twilio Agent Connect is presented as a self-hosted Python and TypeScript SDK under the MIT License, with voice streaming and integrations for existing contact-center workflows.
Claude Code often spends frontier-model tokens on file I/O and predictable code generation rather than reasoning, motivating a routing setup built with Portal by Spotify’s AiKA Modes. The implementation defines two public modes—bulk-reader for multi-file analysis and code-writer for pattern-based generation—both shown with Gemini 2.5 Flash, while a Claude Code plugin named shunt enforces delegation through hooks, scripts, and skills. PreToolUse hooks block large Read and shell-based file reads above a configurable threshold, while scripts send files or references to ephemeral worker invocations without exposing the full file corpus or generated code to Claude’s context. In a Java monorepo benchmark, mean bulk-read savings were around 90%, although code-write savings were harder to measure. The approach does not suit editing or reasoning tasks: worker summaries lacked reliable line numbers, missed a subtle thread-safety bug, and added 10–30 seconds of latency, so targeted reads and Claude’s reasoning remain necessary.
Claude Code often spends frontier-model tokens on file I/O rather than reasoning, prompting a routing approach that sends predictable work to cheaper models. The post implements this with two public AiKA Modes in Spotify’s Backstage Portal: bulk-reader summarizes large file sets, and code-writer generates patterned files from a specification and reference file, using Gemini 2.5 Flash in the examples. A Claude Code plugin called shunt enforces delegation through PreToolUse hooks, scripts, and skills; reads above a configurable threshold, defaulting to 350 lines, are blocked and redirected. In tests on a Java monorepo, bulk-read saved around 90% of Claude’s tokens, while generated code can go directly to disk. The design does not cover editing or reasoning, and 10–30-second delegation latency makes small tasks counterproductive.
Uber describes a Software Factory in which AI tools and managed agents support software development, with more than 70% of pull requests attributed to local or cloud agents and over 3,600 agent skills. From February to August 2026, weekly active users grew sevenfold and weekly agentic requests 9.4-fold, while holding a model constant showed cost per 1,000 requests down almost 34% and cost per session down 52% from its June peak. The framework organizes usage into four layers and decomposes session spend into adoption, engagement, input and output workload, token pricing, and related execution factors measured through portfolio, model, driver, and managed-agent outcome metrics. Its optimization program uses real-work benchmarks, Pareto model selection, cheaper subagents, prompt caching, graph-grounded context, batched tool calls, and session-level waste analysis to reduce zero-value token use while improving or maintaining quality.
The DeepSWE benchmark comparison between DeepSeek V4 Pro 0813 and Claude Fable 5 reveals a trade-off between cost and accuracy. Claude Fable 5 leads on the first attempt with 69.7% pass@1 versus DeepSeek V4 Pro’s 62.8%, but costs 90 times more per rollout ($21.63 vs $0.24). Under repeated attempts, DeepSeek V4 Pro surpasses Fable, achieving 88.5% pass@4 compared to Fable’s 84.1%. The two models show unusually low per-task correlation (0.39), meaning they fail on different tasks. A cascade strategy that runs DeepSeek V4 Pro first and escalates to Claude Fable 5 only on failures solves 82.7% of tasks at $8.28 per task, outperforming either model used alone.
Unity AI Gateway’s Smart Routing, now in Beta, addresses the cost and choice overload of assigning coding tasks to increasingly diverse models and harnesses. It uses task-aware routing: a small, low-latency model classifies the initial task description and metadata, then a policy selects a cheaper, medium, or frontier model; Omnigent can also choose the coding harness and route sub-agents. The design keeps consecutive turns on one model to preserve cache-hit rates, while escalating complex work and delegating simpler tasks to lower-cost options. On internal coding workloads, Smart Routing achieved 35% savings and outperformed any single model at 65% of Opus 5’s cost; public benchmarks reported matched Opus 5 performance with 56% savings. The team identifies mid-session reassessment, real-session evaluation, and affordable switching as continuing challenges.
AI coding tools have improved velocity metrics at Databricks, but broad deployment creates exponentially growing costs that can threaten the efficiency gains they deliver. The post presents a cost-management playbook aimed at preserving broad, low-friction access while keeping aggregate spending within a roughly fixed per-user envelope. It emphasizes the coding-model efficiency frontier, automated evaluations of internal workloads, model-flexible harnesses or meta-harnesses, and request- or task-level routing to the cheapest capable model. Databricks reports that Unity AI Gateway Smart Router reduced average task cost by more than 30% while roughly matching the most expensive model’s quality, and that harness and caching changes cut generated tokens and associated costs by almost 50% without observed quality degradation. The proposed AI Gateway centralizes model access, budget policies, tool configuration, and coding-session traces for ongoing optimization.
Kimi K3 from Moonshot AI is now available on Databricks through the Foundation Model API and Unity AI Gateway, giving customers native access to an open-weight model alongside proprietary models from Anthropic, OpenAI, and Google Gemini. The announcement positions Kimi K3 as a frontier-quality option for coding, agentic reasoning, and document understanding, with deployment flexibility, customizability, and zero data retention coverage. Databricks says the model runs against governed enterprise data through Unity Catalog, while Unity AI Gateway provides permissions, safety guardrails, audit trails, spend controls, routing, and a standardized REST API within Databricks’ security perimeter. The post cites a score of 57 and rank of #4 in Artificial Analysis’ Intelligence Index, internal benchmarks showing performance at the level of leading proprietary models, and 50–72% lower cost per task; availability includes US hosting for AWS and GCP workspaces.
Cursor Router selects models for developer turns using production behavior rather than benchmark scores, balancing user satisfaction and cost. Its routing system first uses Compass, a continuous 0-to-1 complexity predictor trained on follow-up behavior, to keep simple turns on a price-efficient model or upgrade complex ones. Then a taxonomy of domains, tasks, and modifiers compares frontier-model strengths, while an optimizer applies a one-sided 75% uplift threshold and each mode's cost budget. The dataset contains hundreds of thousands of live Cursor turns, preserving conversation signals, model-switching effects, inferred performance from subsequent user actions, and token-based costs including cache misses. Current results report Auto Intelligence at above Fable-level satisfaction with 68% lower cost, and Auto Balance outperforming Opus 4.8 at 41% lower cost while raising satisfaction by 3%.
Unity AI Gateway is generally available as a centralized way for enterprises to manage AI costs, security, governance, and model choice across agents and other AI assets. It positions Unity AI Gateway alongside Unity Catalog: Unity Catalog supplies identity, permissions, lineage, and auditing, while the gateway applies runtime guardrails and contextual policies across AI interactions. The gateway centralizes usage data in Unity Catalog, offering dashboards, Genie-powered analysis, granular cost attribution, budgets, hard spend caps, and a single-query API for frontier models from providers including Anthropic, OpenAI, Gemini, Kimi, and GLM. Smart Routing is in beta and dynamically selects models using quality, cost, performance, availability, and budget; the announcement says customers already use the gateway at scale, including more than one quadrillion tokens passing through it in the past year.
Google Cloud API Gateway adds model routing in Public Preview for AI applications that need to direct traffic among models without hardcoded endpoints or open-source proxies. The serverless ingress layer accepts OpenAI-compatible requests and uses routing rules in an OpenAPI 3.x specification, where x-google-api-management defines backends and routers map virtual model names to targets such as Gemini, Claude, or OpenAI OSS-GPT. After deployment, clients call a stable endpoint; the Gateway transcodes the request to the backend’s native schema, adds an Agent Platform authentication token, and forwards it, separating client authentication from provider credentials. The guide also notes that every backend in one router must share a host, and shows standalone routing alongside possible Agent Gateway integration for security governance.
Retail and CPG joint planning meetings often spend time reconciling fragmented, stale, and inconsistent data instead of deciding how to adjust spend, inventory, or forecasts. The proposed Monday Morning Brief replaces the manually stitched weekly deck with an overnight agent-generated brief built from continuously refreshed point-of-sale, shipment, and inventory data alongside trade, promotion, forecast, and external signals. It uses Delta Sharing for a shared view, Genie Ontology for business context, Unity AI Gateway for permissions, guardrails, logging, rate limits, and human approval, and supports models across AWS, Azure, and Google Cloud. The agent scans item-store combinations, ranks material watchouts, answers cited natural-language follow-ups, and can draft actions such as purchase-order changes, while humans retain approval for consequential decisions. The proposed progression from report to ritual to intelligent decision system includes a 90-minute workshop, a day-30 pilot scope, and a first live Monday targeted for day 90.
GPT-5.6 Sol and Kimi K3 were compared on the DeepSWE software engineering benchmark using 904 graded rollouts across 113 tasks. Sol wins single-shot pass@1 (72.7% vs 68.5%) and reliability (84.5% tasks solved four-for-four versus 76.6%), while Kimi K3 leads pass@4 (89.4% vs 85.8%) with wider coverage at a substantially lower cost—$4.65 per rollout versus Sol’s $8.37, delivering 14.7 solves per $100 against Sol’s 5.3. The two models have a low task correlation of 0.46 and disagree on 18 tasks, enabling a cascade routing strategy that runs Kimi first and escalates to Sol when the test suite rejects the result, achieving 85.6% accuracy, beating both models individually and a perfect one-shot oracle router. Kimi K3 is open-weight and available on Together AI.
Cursor Router now powers Auto mode, analyzing each request by task type and complexity before sending it to a suitable model. Frontier models handle demanding work, while price-efficient models handle other requests, with Cost, Balance, and Intelligence modes defining different positions on the cost-intelligence pareto frontier. Cost optimizes token spend, Balance targets strong quality, and Intelligence targets frontier quality; Balance and Intelligence bill at the routed model’s rate. Admins can enable routing by team or group, restrict modes, set defaults, allow or block underlying models, use soft or hard enforcement, and choose whether to display the routed model, while the feature is available across desktop, web, iOS, CLI, and the SDK and is on by default for Teams plans.
Cursor is launching Cursor Router, a model-routing system for Teams and Enterprise plans that automatically selects a model for each coding request to target frontier-quality results at lower cost. It classifies requests using query, context, task complexity, domain, and model behavior, trained on more than 600,000 live requests and optimized for user satisfaction (AFC); the classifier is cache-aware and designed for frequent model updates. In online A/B testing across millions of requests, the company reports frontier-quality performance at 60% savings, while early-access accounts saved 30%–50% versus routing all traffic at Opus 4.8 API rates without a quality decrease. Three modes—Intelligence, Balance, and Cost—let teams choose a cost-intelligence tradeoff, with administrators controlling defaults, availability, and allowed models across supported Cursor clients.
The best engineering reads, once a week.
