What Is an AI Factory? Power In, Tokens Out
An AI factory is a fixed power and cooling budget converted into model output. This explainer maps its four parts and the throughput-per-watt trade-offs that connect them.
Loading…
Topic
Inference
49 posts about Inference. Every summary links to the original.
- 49 posts
- 13 companies
- 3 explainers
See the idea
Inference explainers
How Blackwell Serving Turns Peak FLOPs Into Cheap Tokens
Blackwell inference is a pipeline, not a chip upgrade. This explainer walks five stages — fleet validation, batching, KV caching, low-precision kernels, and speculative decoding — that turn peak FLOPs into cheap tokens.
How BioNeMo Turns a Target List Into Folded Structures
BioNeMo Inference Runtime runs one pipeline per target: parse, tokenize, generate features, infer on a GPU replica, and write a structure. This diagram traces those stages and the throughput trade-off.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
Speeding up LLM inference with P-EAGLE in vLLM Speculators
P-EAGLE (Parallel EAGLE), implemented in Speculators v0.6.0, extends EAGLE-3 with parallel drafting so a draft model predicts multiple future tokens in one forward pass rather than sequentially. Speculative decoding pairs this small draft model with a verifier model, which checks the proposed sequence in parallel and accepts only its correct prefix while preserving the verifier’s output distribution. The post adds Conditional Drop-token (COD) sampling, learnable mask embeddings and hidden states, and a custom flex-attention mask to support multi-depth prediction within the EAGLE-3 framework. The post presents an end-to-end workflow covering data preparation, vLLM hidden-state extraction, training and deployment, including a Qwen3-8B example that takes about 50 minutes on four H100s. A published Qwen3-a8B draft model reports an average length of 3.044 across the listed datasets.
NVIDIA PAIR Virtual Inference Router Expands Available Compute on Your Local Network
NVIDIA Personal AI Router (PAIR) addresses the GPU bottleneck created when multi-agent workflows and concurrent local sessions generate many independent inference requests. It acts as a virtual inference router rather than an inference engine, proxying compatible Ollama and LM Studio interfaces, discovering and securely pairing local systems, and scheduling each request onto one eligible node based on readiness, engine state, model availability, workload, and GPU utilization. The agent keeps one familiar connection, while PAIR streams responses back and exposes placement through Jobs and metrics views; it does not merge GPUs, pool VRAM, or split a single request across machines. In an unofficial Hermes five-subagent demonstration, a three-device cluster averaged 8 minutes 48 seconds versus 18 minutes on one RTX Spark laptop, with results explicitly dependent on configuration and workload.
Co-Designing AI Models Using Speculative Decoding for Faster LLM Inference
Speculative decoding accelerates large language model inference by having a smaller draft model propose multiple tokens and a target model verify them in parallel, preserving the standard output sequence under its acceptance criteria. It reduces decoding iterations and increases target-model arithmetic intensity, but speedup depends on acceptance length, verification time, draft latency, batch size, and hardware behavior. For attention-heavy workloads, the recommended starting point is D = 128/G - 1, where G is query heads per KV head; larger D values should align G × (1 + D) with 128-token kernel tiles. The remaining guidance covers compute-bound GEMMs, low-latency draft-cost tradeoffs, and choosing among external models, EAGLE-3, MTP, DFlash, DSpark, and suffix or n-gram methods; SPEED-Bench, TensorRT LLM, and NVIDIA/Model-Optimizer support measurement, training, and quantization workflows.
Enterprise-Grade Precision for Long-Context Multimodal Embedding Inference on Cloud TPU
Embedding models convert text, images, and audio into dense vectors used for semantic search, recommendation, intent classification, content discovery, and clustering. The post describes production support for long-context Qwen3 Embedding models on Cloud TPUs through native vLLM TPU integration and GKE Custom Compute Classes, allowing prioritized autoscaling across TPU and GPU capacity pools. Its implementation pads vocabulary tensors for hardware-safe sharding, hardens lazy loading, pre-warms sharding-aware JAX/XLA compilation caches, and uses a hybrid StepPool with CachedRequestState to preserve pooling state during chunked prefill and preemption. Parity evaluations target cosine similarity of at least 0.999 for text and 0.995 for multimodal inputs; on TPU Ironwood, Qwen3-Embedding-8B reached 83,996 total tokens per second and 5.13 requests per second at 16K+ sequence length with bf16 and TP=4.
A/B test models in production
A/B testing model changes in production by sending fixed shares of live endpoint traffic to one control and up to 20 variants, allowing teams to compare user-facing outcomes rather than relying only on shadow-traffic checks. The endpoint router keeps variants out of the base traffic split, resamples requests that reach the control, and treats experiment percentages as absolute shares independent of replica counts or autoscaling. A 95/5 experiment can be updated to 80/20 and then 50/50 by resending the complete member list; an etag rejects stale concurrent updates. Measurement joins per-deployment platform metrics with product signals using the deployment ID in response metadata. To finish, a winning variant is promoted through a blue-green rollout before deleting the experiment, while deletion alone returns traffic to the control and can unwind a losing test.
DeepSeek-V4 Flash 0731 vs GPT-5.6 Luna on DeepSWE: Cost and Coding
On the DeepSWE coding benchmark, GPT-5.6 Luna achieves 67.2% pass@1 versus DeepSeek-V4 Flash’s 53.3%, a 14-point gap. DeepSeek costs $0.10 per task, roughly six times less than Luna’s $0.61, delivering 532 solves per $100 versus Luna’s 110. A cascade that runs DeepSeek first and escalates only on failure solves 78.9% of tasks at $0.385 each, beating Luna’s solo accuracy and cutting cost by 37%. DeepSeek fails more gracefully (9% broken tests vs 15%) but is slower and dramatically weaker on JavaScript (35% vs 60%). Luna dominates seven of eight task domains, while DeepSeek leads only on query and config languages, illustrating how inexpensive models can combine with flagships for superior price-performance.
Together AI announces strategic partnership with Moonshot AI to natively serve Kimi models
Together AI announces a strategic partnership with Moonshot AI to serve Kimi models as a launch platform, beginning with Kimi K3 and extending to future open-weights releases. Kimi K3 is described as a 2.8-trillion-parameter sparse Mixture-of-Experts model with native vision and a 1M-token context window, using Kimi Delta Attention, Attention Residuals, per-head Muon, and quantile-based expert load balancing. Together says developers can access the models through Serverless, Provisioned Throughput, and Dedicated Inference, with US-hosted infrastructure, zero data retention, autoscaling, and a 99% uptime SLA for Provisioned Throughput. The partnership also supports post-training through full-weight and LoRA reinforcement learning and supervised fine-tuning, with checkpoints deployable directly to inference; Kimi K3 is available at launch, while future Moonshot releases are planned for the same platform.
ThunderAgent: 2x Faster Agentic Inference for Synthetic Data Generation at Scale
ThunderAgent is a program-level scheduler for high-throughput agentic inference, developed by researchers at Georgia Tech, UIUC, CMU, and Together AI. It solves KV cache thrashing, which occurs when request-level engines like vLLM and SGLang evict an agent’s KV cache during tool calls and recompute it on resume, creating a cascade of evictions. ThunderAgent tracks each workflow as a schedulable program, pausing low-priority workflows under memory pressure and routing resumed programs through a global waiting queue to the node with the most available capacity. In synthetic data generation pipelines, it achieves up to 2.5x single-node throughput and 2.4x speedup on 8 nodes with near-linear scaling, and was accepted as an ICML 2026 Spotlight paper.
Kimi K3 by Moonshot now available on Modal
Moonshot released Kimi K3, a 2.8 trillion parameter multimodal mixture-of-experts transformer with a 1 million token context window and native vision, now on Modal. Modal runs the model at 460 tokens per second on release day through a partnership with Moonshot and vLLM, with token-based pricing on a Shared API or dedicated Auto Endpoint. K3 activates 16 of 896 experts per token and uses Kimi Delta Attention along with Attention Residuals to achieve roughly 2.5 times the scaling efficiency of its predecessor K2. Moonshot employed quantization-aware training from the SFT stage with MXFP4 weights and MXFP8 activations, and contributed a new prefix caching implementation to vLLM to address challenges from KDA. Modal deployed a custom DFlash speculative decoding drafter tuned to K3’s architecture, yielding 360 percent faster interactivity and 88 percent higher throughput on agentic workloads.
ZeroEntropy is joining Notion
Notion announces it has acquired ZeroEntropy, whose team will join Notion. In less than two years, ZeroEntropy built high-efficiency training and inference for task-specific models using custom kernels, an inference stack, and model-specialization recipes. As a Notion customer, its reranker made unified search up to 30% faster, cut reranking latency by 85%, maintained constant answer quality, and lowered inference cost. ZeroEntropy has also trained zembed-1 and zerank-2 and published the zELO method; founder and CEO Ghita Houir Alami will lead Notion’s newly formed Model Research team. The acquisition supports Notion’s goal of delivering AI that is fast, affordable, and tailored to specific work for businesses of every size.
Together AI and Y Combinator partner to launch the first dedicated GPU cluster for the YC community
Together AI and Y Combinator are partnering to provide YC portfolio companies with a dedicated GPU cluster, addressing compute access as a growing bottleneck for AI-native startups. The service is designed for both inference and training, letting founders reserve, provision, and manage GPUs through Together’s self-service portal rather than making long-term commitments or routing usage through YC. Startups can run short-term sprints at long-term rates, use capacity from single-node workloads to larger deployments, and receive GPUs in minutes, while the cluster is already running at full utilization and companies can plan needs months ahead. Together says its research on attention mechanisms and the Mamba architecture targets faster inference and improved unit economics. The partners plan to expand the cluster and support startups as their requirements grow.
What does 99.9% uptime mean for inference?
The post defines three inference reliability tiers by the failure domains each must survive: 99% requires surviving node-level GPU failures like ECC errors and thermal throttling through automated health checks and fast replica replacement within a single data center. 99.9% requires surviving a full data-center failure by deploying weights across two facilities with enough capacity on each side and routing live traffic to both continuously, not a cold standby. 99.99% requires multi-region deployment with AZ redundancy and reserved failover capacity. The authors argue that vague SLA definitions hide which failure domains are covered, and that infrastructure ownership—whether a provider controls its hardware or rents from a hyperscaler—determines how quickly failures are resolved.
Together AI brings Thinking Machines Lab’s new model Inkling on day 0
Thinking Machines Lab released Inkling, and Together AI announced day-zero access through its production inference platform. The multimodal mixture-of-experts model accepts text, image, and audio inputs and generates text through a unified decoder architecture, with controllable inference effort for trading reasoning depth, token usage, and latency. Inkling has 975B total parameters, 40B active parameters per token, a 1M-token context window, query-conditioned relative attention, short causal convolutions, and shared-sink MoE routing; Together serves it with a FlashAttention-4-based kernel. Preliminary evaluations report strong results across scientific reasoning, mathematics, coding, agentic, vision, and audio benchmarks, while the model is also post-trained for forecasting and calibrated prediction. It is available on Together Serverless through OpenAI-compatible APIs without infrastructure provisioning.
Open, convenient and predictable: Introducing Provisioned Throughput
Companies adopting open-weight models for production workloads have faced a trade-off between convenient but best-effort serverless inference and powerful but complex dedicated GPU instances. Together AI’s Provisioned Throughput fills that gap with reserved inference capacity sold as Provisioned Throughput Units (PTUs). Each PTU is priced at $0.05 per minute and guarantees a fixed rate of tokens per minute for a specific model, with a 99% uptime SLA. Input, cached input, and output tokens draw down capacity at different rates, letting teams optimize for their traffic shape. Available now for MiniMax M3 and GLM-5.2 with capacity in North America and EMEA and a one-month minimum term, at full utilization costs can run roughly $0.36 per million input tokens and $2.16 per million output tokens, up to 90% below proprietary alternatives.
Multi-token Residual Prediction
Multi-Token Residual Prediction (MRP) adapts multi-token prediction for diffusion language models by training a small module to predict the residual between adjacent denoising steps rather than the full next-step distribution. This residual target is easier to learn, allowing a tiny 3-layer transformer to accurately predict corrections across multiple steps. In the static regime, MRP enables lossless speculative decoding with up to 1.56× throughput in SGLang, or direct decoding with a small quality cost. In the dynamic regime, MRP recovers up to +16 accuracy points lost by aggressive low-threshold decoding through a remasking mechanism. The module is trained on a frozen backbone and composes cleanly with existing inference methods.
Announcing our $800M Series C to accelerate the shift to open-source AI
Together AI announces an $800M Series C to accelerate its open-source AI platform as production demand for inference grows. The company says the round includes $800 million from investors including Aramco Ventures, NVIDIA, Vista Equity, General Catalyst, and others, plus commitments for more than 500 MW of compute capacity to be capitalized independently by new investors. Together argues that closed frontier-model economics become unsustainable as agents and workflows scale, while open-weights models such as DeepSeek, Nemotron, MiniMax, Kimi, and GLM can provide equal or better performance at lower cost. Its full-stack approach spans models, kernels, compilers, inference systems, training infrastructure, hardware utilization, and software, with cited work including FlashAttention-4, Together Megakernel, together.compile, and expanded post-training APIs for tool calling, reasoning, and vision-language models.
Together AI at ICML 2026: frontier research across the full stack
Together AI says nine papers from its researchers and peers were accepted to ICML 2026 in Seoul, spanning frontier agents, model shaping, algorithmic and systems optimization, and kernels. The collection is presented as a full-stack view of AI research, with work ranging from DSGym’s unified evaluation and training API for more than 1,000 data-science tasks to ThunderAgent’s program-aware scheduling, TTT-Discover’s test-time reinforcement learning, and RARO’s verifier-free adversarial training. Reported results include up to 3.6× agent throughput, state-of-the-art discoveries from an open 120B model, a 25% expert-response win rate without a verifier, and 5M-token context on one node for Untied Ulysses. Aurora illustrates the research-to-production loop: its adaptive speculative-decoding work ships as the ATLAS speculator on the Together platform.
Achieve state-of-the-art inference latencies with speculative decoding
Modal launched Auto Endpoints, combining speculative decoding with Blackwell GPUs, SGLang engine, and Modal Servers to achieve low inference latency for production deployments. The decode phase, where each token requires a full forward pass loading model weights, is the primary latency source, making speculative decoding the most impactful optimization. Speculative decoding runs a separate DFlash speculator model in parallel with the target model using KV projections and a custom Triton kernel, achieving near-linear speedup proportional to acceptance length. In the Decagon voice case study, Modal cut 100ms from a 290ms baseline by mid-training a custom speculator on synthetic data, beating proprietary providers by 60ms. Modal plans to automate speculator training with continuous improvement on production data, moving toward autospec for Auto Endpoints.
Introducing Modal Auto Endpoints: Optimized inference you actually own
Modal Auto Endpoints is a self-serve service for production-grade LLM inference that gives teams ownership of the entire serving stack, not just an API. Unlike proprietary providers, it exposes the inference code, engine flags, and GPU selection, and provides detailed metrics like speculative decoding acceptance length and per-replica token latency quantiles in a dashboard. Built on Modal’s infrastructure, it uses autoscaling to handle traffic spikes by spinning up additional replicas automatically, global GPU capacity without capacity management, and Modal Servers for ultra-low-latency routing with only 5ms overhead. For each supported model, it offers a pre-configured deployment using speculative decoding with the DFlash block-diffusion drafter from Z Lab, and teams can benchmark latency and throughput tradeoffs with a click after deployment. The platform is designed with a declarative interface based on workloads and SLOs, with future automation levels including autoinference, autospec, autodistill, and autoresearch for continuous improvement of inference performance.
Speculation Is All You Need
Speculative decoding is presented as the most important optimization for LLM inference, delivering speedups of 2x-3x rather than small percentage gains from kernel tuning. The technique accelerates the decode phase by using a lightweight draft model to generate candidate tokens that the target model processes in parallel, with lossless resampling. The authors released DFlash speculators for multiple Qwen models, achieving an additional 5-20% speedup over existing baselines, and worked with SGLang to integrate them. On a B200 node, the Qwen 3.5 122B-A10B model with speculation reaches over 1000 tok/s at concurrency 1, compared to 250 tok/s without speculation, demonstrating a 4x speedup. Training these speculators is simplified because the data-generating process is the target model itself, making data collection trivial, and the post envisions a flywheel where custom speculators and distillation enable continuous improvement in latency, cost, and quality for self-hosted inference.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.