Loading…
Topic
27 posts about Model Routing. Every summary links to the original.
See the idea
Antigravity SDK's hybrid architecture separates planning from execution: a cloud model plans, local models run most tokens on-device. See how routing, execution, and the data boundary fit together.
A benchmark can support two different verdicts: pass@1 on the first attempt, or pass@k with retries measured as cost per solved task. This comparison shows which reading matches your workload.
Understand how to choose between frontier models, efficient alternatives, and routing systems by weighing capability gains against token costs and latency constraints.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Ramp describes an internal LLM gateway that processes trillions of tokens daily and uses dynamic failure-aware routing to reduce AI spend. It combines an EWMA of genuine provider-side failure rates with Thompson Sampling over a Normal-Inverse-Gamma posterior for the mean and variance of log-latency. For each request, the router estimates the probability of outright failure or missing the caller's deadline, incorporates model cost, and orders options while preserving caller-defined model preferences and fallbacks. In a reranker with a six-second deadline, the strategy reacted to Gemini 3.1 Flash Lite rate limiting by switching to GPT 5.4 Nano Flex, whose latency matched Standard at half the price. Larger experiments reported more than 25% cost savings without degradation, while a streaming-equivalent deployment for Ramp Inspect produced 30% savings.
Agent swarms are evaluated as a way to scale software work beyond what a single long-running agent can reliably manage. The test revisits building SQLite in Rust from its documentation, comparing old and new swarms with identical models and time budgets against a held-out SQL suite; the new system performed better in every configuration, reaching 80% with Grok 4.5 in four hours. Planners decompose task trees while workers execute narrower pieces, supported by a purpose-built version control system and mechanisms for design conflicts, merge collisions, megafiles, intentional breakage, and review. Reported results include a full-suite pass using 9,908 engine-code lines versus 64,305 for the old harness, while costs ranged from $1,339 for an Opus 4.8 hybrid to $10,565 for GPT-5.5 alone, indicating that cheaper models can perform most execution after frontier planning resolves ambiguity.
AI spending has moved from experimental pilots into a recurring operating expense, reaching $1.5 trillion globally in 2025, while only 39% of organizations in a cited McKinsey study could trace AI investment to enterprise-level EBIT impact. The launch introduces the Cursor CFO Council, a quarterly forum for finance leaders that will develop shared frameworks for AI productivity, returns on intelligence, model allocation, and cost management. Analyses using Cursor data found higher token usage associated with higher median year-over-year revenue growth, while usage rose after model improvements, especially for high-complexity work. Yet adoption and benefits are concentrated: p99 developers produced 46x more AI-assisted lines per day than median active users, and agent-request costs varied nearly ninefold across model families. With usage-based pricing making intelligence a less predictable variable cost, the council will meet beginning in August to advance measurable and efficient AI economics.
The post argues that a large AI bill does not show excessive use: companies can overspend on routine work while using too little AI where advanced models could create value. It proposes measuring work in atomic tasks—such as invoices coded or pull requests reviewed—rather than tokens, with cost defined by tasks attempted and value by successful tasks. The operating model uses defaults that pair routine work with the cheapest model meeting quality benchmarks, medium reasoning, and flexible latency, while escalating ambiguous, high-stakes work to frontier models at higher effort. It also recommends attributing spend by provider, product, team, and workflow, finding concentrated costs, benchmarking repeated tasks, repricing after model releases, and centralizing controls in one gateway. The conclusion is that efficiency should be treated as an engineering achievement, so cheaper routine execution funds ambition on rare tasks where extra intelligence can materially change outcomes.
This release introduces quality-of-life improvements to the Agents Window, centered on full-screen work areas and more compact agent conversations. Full screen expands files, changes, canvases, PRs, browsers, or terminals across the right panel, replaces chat with a floating prompt bar, and is available through the panel control, command palette, or Cmd/Ctrl+Shift+M. Compact chats offer Compact, Balanced, and Detailed tool-call density settings, ranging from concise results with minimal traces to near-complete step-by-step context. The release also improves PR review visibility, scrolling, streaming, context persistence, MCP authentication, model selection, search, and task status, while fixing reliability issues across PRs, background and cloud agents, MCP/OAuth, shortcuts, hooks, model routing, webhooks, and mirrored repositories.
AI token spend is volatile, consumption-based, and distributed across teams, making usage and cost difficult to monitor. Ramp's pipeline ingests LiteLLM and OpenRouter events through authenticated, tenant-scoped webhooks, Kafka, and ClickHouse, then aggregates them for REST dashboards and AI-generated forecasts. LiteLLM uses callbacks with token counts and metadata, while OpenRouter sends OTLP traces; ReplacingMergeTree deduplicates replayed events using business_id, source, and event_id. Customers can connect gateways with an API key, configure LiteLLM callbacks or OpenRouter Broadcast, and pass attribution metadata. The resulting visibility supports breakdowns by provider, model, department, user, project, and use case, and exposed phantom Gemini reasoning tokens after a LiteLLM upgrade, geo pricing premiums, runaway loops, and prompt bloat.
Agent Bricks is presented as Databricks’ enterprise platform for building, deploying, and governing agents that operate on business data under real identities, permissions, and operational constraints. The platform combines multi-model and framework support, execution, routing, fallback, cost optimization, and unified governance through Unity Catalog and AI Gateway, including on-behalf-of token passing and observability across data, models, MCPs, and APIs. Its context layer uses metadata such as schemas, business definitions, lineage, permissions, and data-quality signals, while Genie Spaces, Document Intelligence, Knowledge Assistant, and Agent Mode address structured and unstructured business information. The announcement includes general availability for Document Intelligence, Custom Agents on Apps, and Supervisor Agent, plus AI Gateway guardrails, managed OAuth MCP Connectors, web search, and MLflow’s CLEARS evaluation framework; the post reports 70% higher accuracy than standard RAG and a 30% improvement in multi-step workflows.
The best engineering reads, once a week.
