What Is an AI Factory? Power In, Tokens Out
An AI factory is a fixed power and cooling budget converted into model output. This explainer maps its four parts and the throughput-per-watt trade-offs that connect them.
Loading…
Topic
Inference
49 posts about Inference. Every summary links to the original.
- 49 posts
- 13 companies
- 3 explainers
See the idea
Inference explainers
How Blackwell Serving Turns Peak FLOPs Into Cheap Tokens
Blackwell inference is a pipeline, not a chip upgrade. This explainer walks five stages — fleet validation, batching, KV caching, low-precision kernels, and speculative decoding — that turn peak FLOPs into cheap tokens.
How BioNeMo Turns a Target List Into Folded Structures
BioNeMo Inference Runtime runs one pipeline per target: parse, tokenize, generate features, infer on a GPU replica, and write a structure. This diagram traces those stages and the throughput trade-off.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
vLLM vs. SGLang: Choosing the right inference engine for your workload
Open weight models now process nearly 80% of production tokens on Vercel’s AI Gateway, up from under 10% in December 2025, and OpenRouter shows a similar climb. This surge drives teams to consider self-hosting via inference engines like vLLM and SGLang, since swapping backends usually only requires a new base URL for an OpenAI-compatible API. vLLM’s PagedAttention, introduced by UC Berkeley in 2023, splits the KV cache into fixed-size blocks to eliminate wasted contiguous memory reservation and increase GPU throughput. SGLang’s RadixAttention, from LMSYS in January 2024, stores computed KV cache in a radix tree keyed by token prefix for reuse across related requests like multi-turn chat and agents. Independent benchmarks from Modal show comparable performance on random prompts without prefix sharing, indicating that workload type and hardware should guide engine choice.
ReplaySSM: Rethinking State Storage for Faster Hybrid-Model Decoding
Hybrid models interleave state-space model (SSM) layers with attention layers, but SSM layers can become a decoding bottleneck at serving batch sizes due to memory-bound state updates. ReplaySSM represents the SSM state as a checkpoint plus a buffer of recent inputs, separating output computation from state updates. This reduces state memory traffic by roughly half, simplifies rollback for speculative decoding by avoiding state snapshots, and supports low-precision state reads without propagating rounding error through the recurrence, enabling separate precision for output computation and state updates. Evaluated on Nemotron 3 and Qwen3.5 models from 4B to 120B parameters on H100 and B300 GPUs, ReplaySSM achieves 1.20× to 1.47× faster decode steps at batch size 256, with end-to-end gains growing with batch size. The technique is integrated into TensorRT-LLM, vLLM, SGLang, and FlashInfer, and runs in production in GLM-5.3-Flash and Kimi-K3.
Minions: A Dynamic and Distributed Virtual GPU Cloud Built on Idle Apple Silicon Devices
Minions is a dynamic, distributed virtual GPU cloud that aggregates idle Apple Silicon GPUs across macOS devices to run large-scale machine learning inference. The work responds to limits in centralized GPU infrastructure, including high cost, limited elasticity, and long wait times when resources are constrained. Demand for GPU-accelerated machine learning inference keeps rising, while modern Apple Silicon macOS devices carry powerful GPUs that stay largely idle during normal usage. Minions offers flexible GPU inference task execution and provides fault tolerance and efficient resource utilization, letting teams use existing hardware in place of additional GPU procurement. The authors report that idle macOS devices can become a practical distributed GPU inference platform in environments where centralized GPU resources are constrained.
Expanding our enterprise inference capacity with IBM Cloud and NVIDIA
Together AI is collaborating with IBM and NVIDIA to scale enterprise-grade AI inference using a dedicated cluster of NVIDIA B300 GPUs on IBM Cloud with NVIDIA Spectrum-X Ethernet networking. This is the first large-scale inference cluster of its kind on IBM Cloud, and Together AI is the first customer running on it. Together AI operates the inference layer, IBM provides the cloud, and NVIDIA delivers the silicon and networking. The company serves hundreds of trillions of tokens per month to over a million developers, and demand continues to climb. Enterprises choose open models to keep their data sovereign and achieve frontier-level performance at a fraction of closed-model cost.
TypeSafe AI scales Jev to trillions of tokens in three days on Modal
TypeSafe AI’s Jev is a System One Model that outputs structured decisions with probabilities rather than generating text, taking state and questions as input and returning typed values. It uses a custom architecture, a parallel sampler for maximum efficiency, and a training method called Reinforcement Learning for Calibrated Decisions (RLCD). Unlike autoregressive LLMs, Jev produces outputs in parallel, enabling low-latency responses suitable for workloads like large-scale automation, fast browser use, and generative user interfaces. Launched on a Thursday, Jev topped OpenRouter’s leaderboard for contexts between 1K and 10K tokens one week later, handling 15.1% of requests on that platform. To meet demand exceeding one trillion tokens in three days, TypeSafe used Modal’s serverless platform for bursty, high-concurrency inference without managing their own infrastructure, gaining control over networking, routing, and autoscaling needed for ultra-low-latency serving.
Together Link: open models in the harness you already use. Start with one command today.
Together Link connects coding agents to over 40 production-tier open models on Together AI, cutting model spend by over 50% compared to premium closed models. It integrates with Claude Code, Claude Desktop, Codex, OpenCode, and Pi without requiring any workflow changes. Open models such as Kimi K3, GLM 5.3, DeepSeek V4.1 Flash, and GLM 5.3 Flash now handle both complex architectural tasks and everyday fixes at a fraction of the per-token cost. A per-session router reads each session’s first task and routes it to either a fast, low-cost model or a frontier model; when no Anthropic key is provided it routes between GLM 5.3 and GLM 5.3 Flash, and Together AI serves the largest share of OpenRouter tokens for these models. Billing uses the existing Together API key with pay-as-you-go or credit packs, and a per-session tracker shows what the session would have cost on Opus 5.5.
How NVIDIA DSX MaxLPS Maximizes AI Factory Throughput and Efficiency
AI factories often underutilize power capacity due to static provisioning for peak GPU draw. NVIDIA DSX MaxLPS uses policy-governed dynamic power allocation to reclaim stranded capacity. In a joint evaluation with Nscale using Kimi K2.5 workloads on GB300 NVL72 systems, DSX MaxLPS enabled 37% more GPUs (192 vs 140) within the same 264.4 kW budget. Aggregate throughput increased 49.2% to 1.62 million tokens/s, while per-instance throughput remained unchanged and median latency stayed within 5% of baseline. However, P99 time to first token increased 17%, highlighting the need to evaluate tail latency alongside capacity gains.
A kernel-centric path to real-time video generation on Trainium
Reactor and the AWS Neuron Science team collaborated to optimize real-time video generation on AWS Trainium. They focused on autoregressive diffusion models, which combine sequential next-token prediction with iterative diffusion to generate video frames that respond to user input. The teams used the Rolling Forcing model as a proxy, addressing challenges such as dynamic shapes, unusual memory access patterns, and heavy cache management that generic compilers struggle with. Optimizations included splitting SDPA heads across four cores and sequences across two, and batching common components of the diffusion and cache update phases to improve hardware utilization. The collaboration achieved real-time video generation above 16 frames per second on a single Trainium2 chip, a result that generalizes across autoregressive diffusion models.
Accelerating vision-language models with LFM2.5-VL-DSpark
Liquid AI released an experimental DSpark draft model for their vision-language model LFM2.5-VL-3B. The drafter adds a speculative decoding path that trades a minimal increase in memory footprint for a larger speedup without changing output quality. The 280M-parameter drafter, a 4-layer attention-only transformer, captures the target model’s hidden states at fixed tapped layers to draft a block of candidate tokens. On an M5 Max, decoding runs 2.30x to 3.13x faster by task, with end-to-end latency improvements of 1.56x to 2.62x. The drafter ships with day-one support for llama.cpp, MLX-VLM, and SGLang, and is available on Hugging Face in Safetensors and GGUF formats.
Hitting a billion tokens per minute on one GPU by combining a query planner and an inference engine
Modal and CMU built Quail, a query-aware inference layer for AI-SQL that combines a query planner with an inference engine to process structured LLM transformations of database entries. On a multi-join query, Quail achieves over a billion tokens per minute per H100 GPU, over 10x faster than vLLM, costing under 6 cents per billion tokens on Modal. The gain comes from using the query plan to order requests for better KV cache management, including a revised Hydragen-style cascade attention and batching small requests to reduce host overhead. On a new AI-SQL benchmark, Quail runs 1.84x faster than vLLM, but falls behind on agent trace processing. The authors see this as part of a trend toward custom inference engines for specific workloads, with a reusable core like databases post-DataFusion, and note that much work was done by prompting coding agents.
How to serve trillions of tokens for trillion-parameter coding agents
Serving inference for coding agents requires trillion-parameter sequence models like Moonshot AI’s Kimi K2.6 to process trillions of input and output tokens at extremely high relative and absolute performance. To be economically viable, these services must achieve large fractions of hardware peak performance, using contemporary matrix math accelerators like Tensor Cores at petaFLOP per second scale. Modal optimized by batching multiple sequences together, caching KV state, and patching SGLang with NVFP4 micro-scaling on Blackwell GPUs like the B200. Per-replica performance improved 2.8x per user and 5.6x across users, transforming ruinously expensive into a price-competitive service. One deployment processed hundreds of billions of tokens daily and trillions total, showing that careful engineering and benchmarking are key to scaling inference.
Canary rollouts: upgrade models in production without downtime
Deploying a new model checkpoint in production typically requires a tradeoff: a hard swap exposes all users to regressions at once, while a DIY staged cutover relies on manual traffic shifting. Together AI’s rollout system automates this by moving traffic between source and target deployments through configurable canary steps, such as 5%, 25%, 50%, and 100%. Each step runs health checks before shifting traffic, enforces a propagation wait for routing caches, and drains the source only after traffic has moved. Optional metric gates, like p95 latency, run after each step; if a gate trips, the rollout pauses automatically and waits for human action. In a real canary from Qwen2.5-7B to Qwen3.5-9B, a 137% p95 regression was caught at 10% traffic, allowing reversal with no failed requests.
Simplifying Model Serving Across Multiple GPUs with NVIDIA TensorRT Multi-Device Integration in NVIDIA Dynamo-Triton
NVIDIA TensorRT multi-device inference enables a single TensorRT network to execute across multiple GPUs using NCCL-backed distributed collectives, supported from TensorRT 11.0. Integrated into Dynamo-Triton release 26.07, one model instance can own multiple GPUs and expose a single gRPC endpoint so the client does not coordinate GPU ranks. The post demonstrates this with Cosmos 3 Nano video generation, where the transformer uses Ulysses context parallelism to distribute 44,160 video tokens across up to eight GPUs. End-to-end latency drops from 156.6 seconds on one GPU to 34.2 seconds on eight GPUs, achieving a 6.09x transformer RPC speedup. Teams can trade additional GPU resources for shorter request latency while keeping the application interface and surrounding workflow stable.
TensorRT Edge-LLM Completes the MLPerf Edge Agentic Benchmark 6.4x Faster on Jetson AGX Thor
Edge agentic workloads require repeated tool-calling turns, growing conversation histories, and valid function calls on constrained devices rather than single-prompt responses. On one NVIDIA Jetson AGX Thor Developer Kit, TensorRT Edge-LLM ran Qwen3.6-27B in SingleStream mode and completed the benchmark’s 1,007-turn performance workload in 24 minutes and 36 seconds, versus 2 hours and 37 minutes for the llama.cpp reference run. It reported 52.33 tokens per second, 247.12 ms median time to first token, and 87.94% BFCL overall accuracy. The implementation combines NVFP4 weights and activations, FP8 KV cache, reusable prompt prefixes with recurrent-state restoration, and tree-based multi-token prediction; roughly 96% of prompt tokens used hot cache, while tree-based MTP could achieve about 40% more decoding performance than linear MTP with three draft steps.
Migrating from closed to open source models, Together
The post presents a five-stage playbook for moving production workloads from closed-source models to open-source models: discover, evaluate, adapt, decide, and take the system to production. It argues that managed services, model-agnostic harnesses, gateways, and tooling can reduce migration complexity, potentially making the effort weeks to months rather than months to years. Discovery combines workload definition, relevant benchmark filtering, cost-per-task and runtime analysis, playground testing, and narrowing candidates to two or three models. Evaluation separates accuracy from performance and recommends replaying real traffic against explicit goals for cost, speed, and quality, while adaptation may involve prompts, inference settings, context, and the surrounding harness. The excerpt frames migration as iterative rather than assuming a model passes unchanged.
Dense vs. MoE Models: Active Parameters, Throughput, and When to Choose Each
Dense models activate every parameter for every token, while Mixture-of-Experts (MoE) models route each token through a selected subset of feed-forward experts. This changes the relationship between capacity, memory, and compute: total parameters determine the memory required to load a model, whereas active parameters largely determine per-token work. At equal total size, MoE models often provide higher throughput, especially at batch size one, but routing, simultaneous expert residency, and reduced KV-cache room add serving complexity; latency advantages narrow at high concurrency. The comparison uses Nemotron 3.5 Lightning, a roughly 30B-parameter Mamba-2 plus MoE plus Attention hybrid with 3B active parameters, alongside dense and MoE examples, and concludes that dense models suit simpler serving and fine-tuning while MoE models suit throughput-oriented deployments with manageable memory and operational complexity.
When to Use Encode-Prefill-Decode Disaggregation to Accelerate Multimodal Model Serving
Encode-prefill-decode (EPD) disaggregation separates a multimodal model’s vision encoder from LLM prefill and decode, addressing contention when image or video processing delays language-model work. Using NVIDIA Dynamo, the post compares aggregated serving with colocated encoder workers and a heterogeneous, disaggregated encoder tier, with vision embeddings transferred to PD workers through NIXL over UCX RC/TCP Ethernet. EPD is most useful for media-heavy requests, short-to-medium outputs, smaller or quantized models, and mixed text/multimodal traffic; long decode sequences or light media processing reduce its value. In the reported image-heavy test, colocated and heterogeneous EPD reduced TTFT by 58% and 50%, while the heterogeneous tier served 70% more traffic at the same under-100-ms ITL SLO. Mixed-traffic measurements also reduced mean TTFT for text requests from 92.3 to 53.3 ms.
The Open Source AI Stack
The post presents an open model AI stack for developers moving from closed-source models, focusing on ownership, control, and economics. It organizes the stack into independent layers: model, inference, gateways and routers, harness, and tools such as Skills and MCP. Models generate responses from prompts; large models generally handle ambiguity, multi-step reasoning, and long context better, while small models can be faster and cheaper for tightly specified work. The separation lets developers choose providers and models independently and experiment with newly released models without rebuilding their workflow, while the post uses Kimi K3 and GLM 5.3 Flash to illustrate differences in active parameters, capability, speed, and cost.
Optimize vLLM speculative decoding with FastMTP heads
Autoregressive LLM inference is memory-bandwidth bound because each token requires a full pass over billions of parameters, while speculative decoding can verify several drafted tokens in one pass. Speculators 0.6.0 adapts a single native MTP head with FastMTP-style recursive fine-tuning, addressing the mismatch between teacher-forced training and vLLM’s autoregressive reuse of the head. The training loop fuses each next-token embedding with the current hidden state, predicts the following token, feeds the resulting hidden state back, and weights three steps with normalized exponential decay using β = 0.6. Native mtp.* weights extraction, shared full-vocabulary embeddings and output head, and vLLM-compatible stitching support deployment; on Qwen3-Next-80B-A3B-Instruct evaluated on GSM8K, mean accepted length improved at positions 0–2 from 0.897, 0.719, 0.476 to 0.912, 0.776, 0.616.
Achieving Extreme Efficiency through Specialized GPU Kernel Generation
Production inference systems often use generic GPU kernels even though operation shapes combine static model parameters with request-time dimensions such as token counts. Proteus addresses this mismatch by having agents propose specialized kernels, then applying controlled-reference validation, static checks, isolated repeated timing, hidden tests, and consistency checks before retaining measured winners. Its knowledge layer uses actionable, scoped lessons retrieved through hierarchical tags and hybrid keyword-semantic search, while background jobs handle deeper distillation instead of burdening each attempt. On NVIDIA B200 GPUs with a Triton backend, a Gated DeltaNet packed-decode case for Qwen 3.5 122B produced shape-specific wins, including 1.6× for a Batch=4, Key=128, Value=128 path and 0.018 ms latency for Candidate 030. Across the reported kernels, Proteus achieved 1.8–5.2× speedups over the best available in vLLM, with validation and context management identified as the main difficulties.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.