---
title: "Route AI Agent Workloads Across Models with NVIDIA NeMo Switchyard"
description: "NVIDIA NeMo Switchyard addresses the problem of routing AI-agent work across models with different capabilities, costs, latency, and verbosity profiles rather than sending every request to one model. At runtime, its router evaluates a request, context, policies, and system signals, then directs work to a configured model pool, potentially routing individual agent steps and carrying session state when needed. The provider-agnostic switchyard-libsy SDK separates semantic model targets from provider endpoints, while a reference server accepts OpenAI, Anthropic, and Responses API requests and records routing decisions and outcomes. Available approaches include tuning-free LLM classifier, stage, and escalation routers, alongside tunable and custom algorithms. In a LangChain evaluation, escalation routing cut cost 74% versus a frontier-only baseline with a measured six-point accuracy trade-off, while Cognition’s staged routing reported near-frontier coding performance at approximately 28% lower mean cost."
---

# Route AI Agent Workloads Across Models with NVIDIA NeMo Switchyard

[NVIDIA](https://yomu.fyi/company/nvidia-developer-blog) · Michelle Horton · Aug 11, 2026

**Type:** Explainer

## Summary

NVIDIA NeMo Switchyard addresses the problem of routing AI-agent work across models with different capabilities, costs, latency, and verbosity profiles rather than sending every request to one model. At runtime, its router evaluates a request, context, policies, and system signals, then directs work to a configured model pool, potentially routing individual agent steps and carrying session state when needed. The provider-agnostic switchyard-libsy SDK separates semantic model targets from provider endpoints, while a reference server accepts OpenAI, Anthropic, and Responses API requests and records routing decisions and outcomes. Available approaches include tuning-free LLM classifier, stage, and escalation routers, alongside tunable and custom algorithms. In a LangChain evaluation, escalation routing cut cost 74% versus a frontier-only baseline with a measured six-point accuracy trade-off, while Cognition’s staged routing reported near-frontier coding performance at approximately 28% lower mean cost.

## Context

AI-agent workloads may require classification, reasoning, and routine follow-up steps with different model capabilities. Using the largest model for every request can increase cost and latency, while using a smaller model throughout can reduce quality on complex tasks. Routing decisions also need to account for model capability, cost, latency, infrastructure, workload stage, and error tolerance.

## Approach / What changed

NeMo Switchyard provides an orchestration layer and the provider-agnostic switchyard-libsy SDK for routing requests across configured model targets. It supports routing algorithms including LLM classifier, stage, and escalation routers, as well as tunable and custom algorithms. The system can carry routing state across sessions, preserve context such as tool results or affinity decisions, and expose routing through OpenAI, Anthropic, and Responses API-compatible interfaces.

## Takeaways

- The switchyard-libsy SDK separates semantic model names from provider endpoints, allowing model deployments, endpoints, or providers to change without changing the routing integration.
- The stage router uses recent tool activity to estimate how much model capability an agent needs at each turn, while the LLM classifier can preserve session affinity across later turns.
- In LangChain’s 145-task evaluation, escalation routing reduced cost by 74% versus a frontier-only baseline, sent 7% of calls to the frontier model, and incurred a measured approximate six-point accuracy trade-off.

**Tags:** [Architecture](https://yomu.fyi/topic/architecture), [LLMs](https://yomu.fyi/topic/llm), [Open Source](https://yomu.fyi/topic/open-source), [Performance](https://yomu.fyi/topic/performance)

- Source: [NVIDIA](https://developer.nvidia.com/blog/route-ai-agent-workloads-across-models-with-nvidia-nemo-switchyard)
- Source URL: https://developer.nvidia.com/blog/route-ai-agent-workloads-across-models-with-nvidia-nemo-switchyard
- Ingested by Yomu: 2026-08-28T08:00:55.456Z

[Read original post](https://developer.nvidia.com/blog/route-ai-agent-workloads-across-models-with-nvidia-nemo-switchyard)