---
title: "Inference"
description: "51 posts about Inference, summarised, each linking to the original."
---

# Inference
> 51 posts about Inference, summarised, each linking to the original.

## Articles

### [Optimized support for Open NVIDIA Nemotron 3.5 Super VL](https://yomu.fyi/post/optimized-support-for-open-nvidia-nemotron-3-5-super-vl.md)
- Company: [Red Hat](https://yomu.fyi/company/red-hat.md)
- Author: Eldar Kurtić, Helen Zhao, Charles Hernandez, Dipika Sikka
- Published: Oct 15, 2026
- Type: Announcement

Red Hat AI released optimized FP8 and DSpark speculative decoding checkpoints for the Nemotron 3.5 Super VL 120B multimodal mixture-of-experts (MoE) model. These checkpoints are designed to double inference throughput on NVIDIA Hopper GPUs without compromising output accuracy. FP8 quantization uses LLM Compressor to reduce model weights and activations from BF16 to 8-bit floating point, cutting memory and storage requirements in half while preserving quality. DSpark speculative decoding proposes 8-16 tokens in a single forward pass and uses a lightweight Markov head to introduce dependencies between speculated tokens. Benchmarks on HumanEval, math\_reasoning, and RAG datasets show that each technique delivers meaningful gains individually and combining them outperforms any intermediate configuration at every concurrency level.


### [Welcome FLUX.3 Image Klein in Diffusers](https://yomu.fyi/post/welcome-flux-3-image-klein-in-diffusers.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Aritra Roy Gosthipaty, Sayak Paul, Linoy Tsaban
- Published: Oct 15, 2026
- Type: Announcement

The Black Forest Labs FLUX.3 Image Klein model is now available in Diffusers, combining text-to-image, multi-reference image-to-image, and bounding-box layout control into one pipeline. Two open checkpoints are offered: a distilled version using 16 inference steps without classifier-free guidance for fast generation, and a base version requiring 50 steps with guidance. The distilled checkpoint produces a 1024x1024 image in about two seconds on an H100, and quantized NVFP4 and FP8 weights can be loaded via from\_single\_file to reduce memory. The model is built as a Modular Diffusers pipeline, enabling users to remove, replace, or insert custom components, such as a block that captures intermediate clean-image previews. LoRA fine-tuning scripts for both text-to-image and image editing are included, along with a prompting guide for optimal scene descriptions.


### [vLLM vs. SGLang: Choosing the right inference engine for your workload](https://yomu.fyi/post/vllm-vs-sglang-choosing-the-right-inference-engine-for-your-workload.md)
- Company: [Red Hat](https://yomu.fyi/company/red-hat.md)
- Author: Cedric Clyburn
- Published: Oct 14, 2026
- Type: Explainer

Open weight models now process nearly 80% of production tokens on Vercel’s AI Gateway, up from under 10% in December 2025, and OpenRouter shows a similar climb. This surge drives teams to consider self-hosting via inference engines like vLLM and SGLang, since swapping backends usually only requires a new base URL for an OpenAI-compatible API. vLLM’s PagedAttention, introduced by UC Berkeley in 2023, splits the KV cache into fixed-size blocks to eliminate wasted contiguous memory reservation and increase GPU throughput. SGLang’s RadixAttention, from LMSYS in January 2024, stores computed KV cache in a radix tree keyed by token prefix for reuse across related requests like multi-turn chat and agents. Independent benchmarks from Modal show comparable performance on random prompts without prefix sharing, indicating that workload type and hardware should guide engine choice.


### [ReplaySSM: Rethinking State Storage for Faster Hybrid-Model Decoding](https://yomu.fyi/post/replayssm-rethinking-state-storage-for-faster-hybrid-model-decoding.md)
- Company: [NVIDIA](https://yomu.fyi/company/nvidia-developer-blog.md)
- Author: Michelle Horton
- Published: Oct 12, 2026
- Type: Problem & solution

Hybrid models interleave state-space model (SSM) layers with attention layers, but SSM layers can become a decoding bottleneck at serving batch sizes due to memory-bound state updates. ReplaySSM represents the SSM state as a checkpoint plus a buffer of recent inputs, separating output computation from state updates. This reduces state memory traffic by roughly half, simplifies rollback for speculative decoding by avoiding state snapshots, and supports low-precision state reads without propagating rounding error through the recurrence, enabling separate precision for output computation and state updates. Evaluated on Nemotron 3 and Qwen3.5 models from 4B to 120B parameters on H100 and B300 GPUs, ReplaySSM achieves 1.20× to 1.47× faster decode steps at batch size 256, with end-to-end gains growing with batch size. The technique is integrated into TensorRT-LLM, vLLM, SGLang, and FlashInfer, and runs in production in GLM-5.3-Flash and Kimi-K3.


### [Minions: A Dynamic and Distributed Virtual GPU Cloud Built on Idle Apple Silicon Devices](https://yomu.fyi/post/minions-a-dynamic-and-distributed-virtual-gpu-cloud-built-on-idle-appl.md)
- Company: [Apple](https://yomu.fyi/company/apple.md)
- Published: Oct 12, 2026
- Type: Problem & solution

Minions is a dynamic, distributed virtual GPU cloud that aggregates idle Apple Silicon GPUs across macOS devices to run large-scale machine learning inference. The work responds to limits in centralized GPU infrastructure, including high cost, limited elasticity, and long wait times when resources are constrained. Demand for GPU-accelerated machine learning inference keeps rising, while modern Apple Silicon macOS devices carry powerful GPUs that stay largely idle during normal usage. Minions offers flexible GPU inference task execution and provides fault tolerance and efficient resource utilization, letting teams use existing hardware in place of additional GPU procurement. The authors report that idle macOS devices can become a practical distributed GPU inference platform in environments where centralized GPU resources are constrained.


### [Expanding our enterprise inference capacity with IBM Cloud and NVIDIA](https://yomu.fyi/post/expanding-our-enterprise-inference-capacity-with-ibm-cloud-and-nvidia.md)
- Company: [Together AI](https://yomu.fyi/company/together-ai.md)
- Published: Oct 6, 2026
- Type: Announcement

Together AI is collaborating with IBM and NVIDIA to scale enterprise-grade AI inference using a dedicated cluster of NVIDIA B300 GPUs on IBM Cloud with NVIDIA Spectrum-X Ethernet networking. This is the first large-scale inference cluster of its kind on IBM Cloud, and Together AI is the first customer running on it. Together AI operates the inference layer, IBM provides the cloud, and NVIDIA delivers the silicon and networking. The company serves hundreds of trillions of tokens per month to over a million developers, and demand continues to climb. Enterprises choose open models to keep their data sovereign and achieve frontier-level performance at a fraction of closed-model cost.


### [TypeSafe AI scales Jev to trillions of tokens in three days on Modal](https://yomu.fyi/post/typesafe-ai-scales-jev-to-trillions-of-tokens-in-three-days-on-modal.md)
- Company: [Modal](https://yomu.fyi/company/modal.md)
- Published: Oct 5, 2026
- Type: Announcement

TypeSafe AI’s Jev is a System One Model that outputs structured decisions with probabilities rather than generating text, taking state and questions as input and returning typed values. It uses a custom architecture, a parallel sampler for maximum efficiency, and a training method called Reinforcement Learning for Calibrated Decisions (RLCD). Unlike autoregressive LLMs, Jev produces outputs in parallel, enabling low-latency responses suitable for workloads like large-scale automation, fast browser use, and generative user interfaces. Launched on a Thursday, Jev topped OpenRouter’s leaderboard for contexts between 1K and 10K tokens one week later, handling 15.1% of requests on that platform. To meet demand exceeding one trillion tokens in three days, TypeSafe used Modal’s serverless platform for bursty, high-concurrency inference without managing their own infrastructure, gaining control over networking, routing, and autoscaling needed for ultra-low-latency serving.


### [Together Link: open models in the harness you already use. Start with one command today.](https://yomu.fyi/post/together-link-open-models-in-the-harness-you-already-use-start-with-on.md)
- Company: [Together AI](https://yomu.fyi/company/together-ai.md)
- Published: Oct 5, 2026
- Type: Announcement

Together Link connects coding agents to over 40 production-tier open models on Together AI, cutting model spend by over 50% compared to premium closed models. It integrates with Claude Code, Claude Desktop, Codex, OpenCode, and Pi without requiring any workflow changes. Open models such as Kimi K3, GLM 5.3, DeepSeek V4.1 Flash, and GLM 5.3 Flash now handle both complex architectural tasks and everyday fixes at a fraction of the per-token cost. A per-session router reads each session’s first task and routes it to either a fast, low-cost model or a frontier model; when no Anthropic key is provided it routes between GLM 5.3 and GLM 5.3 Flash, and Together AI serves the largest share of OpenRouter tokens for these models. Billing uses the existing Together API key with pay-as-you-go or credit packs, and a per-session tracker shows what the session would have cost on Opus 5.5.


### [How NVIDIA DSX MaxLPS Maximizes AI Factory Throughput and Efficiency](https://yomu.fyi/post/how-nvidia-dsx-maxlps-maximizes-ai-factory-throughput-and-efficiency.md)
- Company: [NVIDIA](https://yomu.fyi/company/nvidia-developer-blog.md)
- Author: Sarah McKenney
- Published: Sep 28, 2026
- Type: Benchmark

AI factories often underutilize power capacity due to static provisioning for peak GPU draw. NVIDIA DSX MaxLPS uses policy-governed dynamic power allocation to reclaim stranded capacity. In a joint evaluation with Nscale using Kimi K2.5 workloads on GB300 NVL72 systems, DSX MaxLPS enabled 37% more GPUs (192 vs 140) within the same 264.4 kW budget. Aggregate throughput increased 49.2% to 1.62 million tokens/s, while per-instance throughput remained unchanged and median latency stayed within 5% of baseline. However, P99 time to first token increased 17%, highlighting the need to evaluate tail latency alongside capacity gains.


### [A kernel-centric path to real-time video generation on Trainium](https://yomu.fyi/post/a-kernel-centric-path-to-real-time-video-generation-on-trainium.md)
- Company: [AWS](https://yomu.fyi/company/aws.md)
- Author: Stephen Zorio
- Published: Sep 25, 2026
- Type: Problem & solution

Reactor and the AWS Neuron Science team collaborated to optimize real-time video generation on AWS Trainium. They focused on autoregressive diffusion models, which combine sequential next-token prediction with iterative diffusion to generate video frames that respond to user input. The teams used the Rolling Forcing model as a proxy, addressing challenges such as dynamic shapes, unusual memory access patterns, and heavy cache management that generic compilers struggle with. Optimizations included splitting SDPA heads across four cores and sequences across two, and batching common components of the diffusion and cache update phases to improve hardware utilization. The collaboration achieved real-time video generation above 16 frames per second on a single Trainium2 chip, a result that generalizes across autoregressive diffusion models.


### [Accelerating vision-language models with LFM2.5-VL-DSpark](https://yomu.fyi/post/accelerating-vision-language-models-with-lfm2-5-vl-dspark.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: xx, Yuri Khrustalev, Leonie Monigatti, Viviana Márquez
- Published: Sep 24, 2026
- Type: Announcement

Liquid AI released an experimental DSpark draft model for their vision-language model LFM2.5-VL-3B. The drafter adds a speculative decoding path that trades a minimal increase in memory footprint for a larger speedup without changing output quality. The 280M-parameter drafter, a 4-layer attention-only transformer, captures the target model’s hidden states at fixed tapped layers to draft a block of candidate tokens. On an M5 Max, decoding runs 2.30x to 3.13x faster by task, with end-to-end latency improvements of 1.56x to 2.62x. The drafter ships with day-one support for llama.cpp, MLX-VLM, and SGLang, and is available on Hugging Face in Safetensors and GGUF formats.


### [Hitting a billion tokens per minute on one GPU by combining a query planner and an inference engine](https://yomu.fyi/post/hitting-a-billion-tokens-per-minute-on-one-gpu-by-combining-a-query-pl.md)
- Company: [Modal](https://yomu.fyi/company/modal.md)
- Published: Sep 24, 2026
- Type: Problem & solution

Modal and CMU built Quail, a query-aware inference layer for AI-SQL that combines a query planner with an inference engine to process structured LLM transformations of database entries. On a multi-join query, Quail achieves over a billion tokens per minute per H100 GPU, over 10x faster than vLLM, costing under 6 cents per billion tokens on Modal. The gain comes from using the query plan to order requests for better KV cache management, including a revised Hydragen-style cascade attention and batching small requests to reduce host overhead. On a new AI-SQL benchmark, Quail runs 1.84x faster than vLLM, but falls behind on agent trace processing. The authors see this as part of a trend toward custom inference engines for specific workloads, with a reusable core like databases post-DataFusion, and note that much work was done by prompting coding agents.


### [How to serve trillions of tokens for trillion-parameter coding agents](https://yomu.fyi/post/how-to-serve-trillions-of-tokens-for-trillion-parameter-coding-agents.md)
- Company: [Modal](https://yomu.fyi/company/modal.md)
- Published: Sep 23, 2026
- Type: Explainer

Serving inference for coding agents requires trillion-parameter sequence models like Moonshot AI’s Kimi K2.6 to process trillions of input and output tokens at extremely high relative and absolute performance. To be economically viable, these services must achieve large fractions of hardware peak performance, using contemporary matrix math accelerators like Tensor Cores at petaFLOP per second scale. Modal optimized by batching multiple sequences together, caching KV state, and patching SGLang with NVFP4 micro-scaling on Blackwell GPUs like the B200. Per-replica performance improved 2.8x per user and 5.6x across users, transforming ruinously expensive into a price-competitive service. One deployment processed hundreds of billions of tokens daily and trillions total, showing that careful engineering and benchmarking are key to scaling inference.


### [Canary rollouts: upgrade models in production without downtime](https://yomu.fyi/post/canary-rollouts-upgrade-models-in-production-without-downtime.md)
- Company: [Together AI](https://yomu.fyi/company/together-ai.md)
- Published: Sep 22, 2026
- Type: Announcement

Deploying a new model checkpoint in production typically requires a tradeoff: a hard swap exposes all users to regressions at once, while a DIY staged cutover relies on manual traffic shifting. Together AI’s rollout system automates this by moving traffic between source and target deployments through configurable canary steps, such as 5%, 25%, 50%, and 100%. Each step runs health checks before shifting traffic, enforces a propagation wait for routing caches, and drains the source only after traffic has moved. Optional metric gates, like p95 latency, run after each step; if a gate trips, the rollout pauses automatically and waits for human action. In a real canary from Qwen2.5-7B to Qwen3.5-9B, a 137% p95 regression was caught at 10% traffic, allowing reversal with no failed requests.


### [Simplifying Model Serving Across Multiple GPUs with NVIDIA TensorRT Multi-Device Integration in NVIDIA Dynamo-Triton](https://yomu.fyi/post/simplifying-model-serving-across-multiple-gpus-with-nvidia-tensorrt-mu.md)
- Company: [NVIDIA](https://yomu.fyi/company/nvidia-developer-blog.md)
- Author: Tanya Lenz
- Published: Sep 21, 2026
- Type: Announcement

NVIDIA TensorRT multi-device inference enables a single TensorRT network to execute across multiple GPUs using NCCL-backed distributed collectives, supported from TensorRT 11.0. Integrated into Dynamo-Triton release 26.07, one model instance can own multiple GPUs and expose a single gRPC endpoint so the client does not coordinate GPU ranks. The post demonstrates this with Cosmos 3 Nano video generation, where the transformer uses Ulysses context parallelism to distribute 44,160 video tokens across up to eight GPUs. End-to-end latency drops from 156.6 seconds on one GPU to 34.2 seconds on eight GPUs, achieving a 6.09x transformer RPC speedup. Teams can trade additional GPU resources for shorter request latency while keeping the application interface and surrounding workflow stable.


### [TensorRT Edge-LLM Completes the MLPerf Edge Agentic Benchmark 6.4x Faster on Jetson AGX Thor](https://yomu.fyi/post/tensorrt-edge-llm-completes-the-mlperf-edge-agentic-benchmark-6-4x-fas.md)
- Company: [NVIDIA](https://yomu.fyi/company/nvidia-developer-blog.md)
- Author: Elizabeth Goodman
- Published: Sep 16, 2026
- Type: Benchmark

Edge agentic workloads require repeated tool-calling turns, growing conversation histories, and valid function calls on constrained devices rather than single-prompt responses. On one NVIDIA Jetson AGX Thor Developer Kit, TensorRT Edge-LLM ran Qwen3.6-27B in SingleStream mode and completed the benchmark’s 1,007-turn performance workload in 24 minutes and 36 seconds, versus 2 hours and 37 minutes for the llama.cpp reference run. It reported 52.33 tokens per second, 247.12 ms median time to first token, and 87.94% BFCL overall accuracy. The implementation combines NVFP4 weights and activations, FP8 KV cache, reusable prompt prefixes with recurrent-state restoration, and tree-based multi-token prediction; roughly 96% of prompt tokens used hot cache, while tree-based MTP could achieve about 40% more decoding performance than linear MTP with three draft steps.


### [Migrating from closed to open source models, Together](https://yomu.fyi/post/migrating-from-closed-to-open-source-models-together.md)
- Company: [Together AI](https://yomu.fyi/company/together-ai.md)
- Published: Sep 16, 2026
- Type: Tutorial

The post presents a five-stage playbook for moving production workloads from closed-source models to open-source models: discover, evaluate, adapt, decide, and take the system to production. It argues that managed services, model-agnostic harnesses, gateways, and tooling can reduce migration complexity, potentially making the effort weeks to months rather than months to years. Discovery combines workload definition, relevant benchmark filtering, cost-per-task and runtime analysis, playground testing, and narrowing candidates to two or three models. Evaluation separates accuracy from performance and recommends replaying real traffic against explicit goals for cost, speed, and quality, while adaptation may involve prompts, inference settings, context, and the surrounding harness. The excerpt frames migration as iterative rather than assuming a model passes unchanged.


### [Dense vs. MoE Models: Active Parameters, Throughput, and When to Choose Each](https://yomu.fyi/post/dense-vs-moe-models-active-parameters-throughput-and-when-to-choose-ea.md)
- Company: [NVIDIA](https://yomu.fyi/company/nvidia-developer-blog.md)
- Author: Elizabeth Goodman
- Published: Sep 15, 2026
- Type: Explainer

Dense models activate every parameter for every token, while Mixture-of-Experts (MoE) models route each token through a selected subset of feed-forward experts. This changes the relationship between capacity, memory, and compute: total parameters determine the memory required to load a model, whereas active parameters largely determine per-token work. At equal total size, MoE models often provide higher throughput, especially at batch size one, but routing, simultaneous expert residency, and reduced KV-cache room add serving complexity; latency advantages narrow at high concurrency. The comparison uses Nemotron 3.5 Lightning, a roughly 30B-parameter Mamba-2 plus MoE plus Attention hybrid with 3B active parameters, alongside dense and MoE examples, and concludes that dense models suit simpler serving and fine-tuning while MoE models suit throughput-oriented deployments with manageable memory and operational complexity.


### [When to Use Encode-Prefill-Decode Disaggregation to Accelerate Multimodal Model Serving](https://yomu.fyi/post/when-to-use-encode-prefill-decode-disaggregation-to-accelerate-multimo.md)
- Company: [NVIDIA](https://yomu.fyi/company/nvidia-developer-blog.md)
- Author: Tanya Lenz
- Published: Sep 9, 2026
- Type: Explainer

Encode-prefill-decode (EPD) disaggregation separates a multimodal model’s vision encoder from LLM prefill and decode, addressing contention when image or video processing delays language-model work. Using NVIDIA Dynamo, the post compares aggregated serving with colocated encoder workers and a heterogeneous, disaggregated encoder tier, with vision embeddings transferred to PD workers through NIXL over UCX RC/TCP Ethernet. EPD is most useful for media-heavy requests, short-to-medium outputs, smaller or quantized models, and mixed text/multimodal traffic; long decode sequences or light media processing reduce its value. In the reported image-heavy test, colocated and heterogeneous EPD reduced TTFT by 58% and 50%, while the heterogeneous tier served 70% more traffic at the same under-100-ms ITL SLO. Mixed-traffic measurements also reduced mean TTFT for text requests from 92.3 to 53.3 ms.


### [The Open Source AI Stack](https://yomu.fyi/post/the-open-source-ai-stack.md)
- Company: [Together AI](https://yomu.fyi/company/together-ai.md)
- Published: Sep 9, 2026
- Type: Explainer

The post presents an open model AI stack for developers moving from closed-source models, focusing on ownership, control, and economics. It organizes the stack into independent layers: model, inference, gateways and routers, harness, and tools such as Skills and MCP. Models generate responses from prompts; large models generally handle ambiguity, multi-step reasoning, and long context better, while small models can be faster and cheaper for tightly specified work. The separation lets developers choose providers and models independently and experiment with newly released models without rebuilding their workflow, while the post uses Kimi K3 and GLM 5.3 Flash to illustrate differences in active parameters, capability, speed, and cost.


[Older posts](https://yomu.fyi/topic/inference/page/2.md)