---
title: "Inference"
description: "51 posts about Inference, summarised, each linking to the original."
---

# Inference
> 51 posts about Inference, summarised, each linking to the original.

## Articles

### [Optimize vLLM speculative decoding with FastMTP heads](https://yomu.fyi/post/optimize-vllm-speculative-decoding-with-fastmtp-heads.md)
- Company: [Red Hat](https://yomu.fyi/company/red-hat.md)
- Author: Rahul Tuli
- Published: Sep 8, 2026
- Type: Tutorial

Autoregressive LLM inference is memory-bandwidth bound because each token requires a full pass over billions of parameters, while speculative decoding can verify several drafted tokens in one pass. Speculators 0.6.0 adapts a single native MTP head with FastMTP-style recursive fine-tuning, addressing the mismatch between teacher-forced training and vLLM’s autoregressive reuse of the head. The training loop fuses each next-token embedding with the current hidden state, predicts the following token, feeds the resulting hidden state back, and weights three steps with normalized exponential decay using β = 0.6. Native mtp.\* weights extraction, shared full-vocabulary embeddings and output head, and vLLM-compatible stitching support deployment; on Qwen3-Next-80B-A3B-Instruct evaluated on GSM8K, mean accepted length improved at positions 0–2 from 0.897, 0.719, 0.476 to 0.912, 0.776, 0.616.


### [Achieving Extreme Efficiency through Specialized GPU Kernel Generation](https://yomu.fyi/post/achieving-extreme-efficiency-through-specialized-gpu-kernel-generation.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Daya Khudia, Leo Li, Lesheng Jin
- Published: Sep 4, 2026
- Type: Explainer

Production inference systems often use generic GPU kernels even though operation shapes combine static model parameters with request-time dimensions such as token counts. Proteus addresses this mismatch by having agents propose specialized kernels, then applying controlled-reference validation, static checks, isolated repeated timing, hidden tests, and consistency checks before retaining measured winners. Its knowledge layer uses actionable, scoped lessons retrieved through hierarchical tags and hybrid keyword-semantic search, while background jobs handle deeper distillation instead of burdening each attempt. On NVIDIA B200 GPUs with a Triton backend, a Gated DeltaNet packed-decode case for Qwen 3.5 122B produced shape-specific wins, including 1.6× for a Batch=4, Key=128, Value=128 path and 0.018 ms latency for Candidate 030. Across the reported kernels, Proteus achieved 1.8–5.2× speedups over the best available in vLLM, with validation and context management identified as the main difficulties.


### [Speeding up LLM inference with P-EAGLE in vLLM Speculators](https://yomu.fyi/post/speeding-up-llm-inference-with-p-eagle-in-vllm-speculators.md)
- Company: [Red Hat](https://yomu.fyi/company/red-hat.md)
- Author: Helen Zhao, Megan Flynn, Dipika Sikka
- Published: Sep 3, 2026
- Type: Explainer

P-EAGLE (Parallel EAGLE), implemented in Speculators v0.6.0, extends EAGLE-3 with parallel drafting so a draft model predicts multiple future tokens in one forward pass rather than sequentially. Speculative decoding pairs this small draft model with a verifier model, which checks the proposed sequence in parallel and accepts only its correct prefix while preserving the verifier’s output distribution. The post adds Conditional Drop-token (COD) sampling, learnable mask embeddings and hidden states, and a custom flex-attention mask to support multi-depth prediction within the EAGLE-3 framework. The post presents an end-to-end workflow covering data preparation, vLLM hidden-state extraction, training and deployment, including a Qwen3-8B example that takes about 50 minutes on four H100s. A published Qwen3-a8B draft model reports an average length of 3.044 across the listed datasets.


### [NVIDIA PAIR Virtual Inference Router Expands Available Compute on Your Local Network](https://yomu.fyi/post/nvidia-pair-virtual-inference-router-expands-available-compute-on-your.md)
- Company: [NVIDIA](https://yomu.fyi/company/nvidia-developer-blog.md)
- Author: Tanya Lenz
- Published: Sep 3, 2026
- Type: Explainer

NVIDIA Personal AI Router (PAIR) addresses the GPU bottleneck created when multi-agent workflows and concurrent local sessions generate many independent inference requests. It acts as a virtual inference router rather than an inference engine, proxying compatible Ollama and LM Studio interfaces, discovering and securely pairing local systems, and scheduling each request onto one eligible node based on readiness, engine state, model availability, workload, and GPU utilization. The agent keeps one familiar connection, while PAIR streams responses back and exposes placement through Jobs and metrics views; it does not merge GPUs, pool VRAM, or split a single request across machines. In an unofficial Hermes five-subagent demonstration, a three-device cluster averaged 8 minutes 48 seconds versus 18 minutes on one RTX Spark laptop, with results explicitly dependent on configuration and workload.


### [Co-Designing AI Models Using Speculative Decoding for Faster LLM Inference](https://yomu.fyi/post/co-designing-ai-models-using-speculative-decoding-for-faster-llm-infer.md)
- Company: [NVIDIA](https://yomu.fyi/company/nvidia-developer-blog.md)
- Published: Sep 2, 2026
- Type: Explainer

Speculative decoding accelerates large language model inference by having a smaller draft model propose multiple tokens and a target model verify them in parallel, preserving the standard output sequence under its acceptance criteria. It reduces decoding iterations and increases target-model arithmetic intensity, but speedup depends on acceptance length, verification time, draft latency, batch size, and hardware behavior. For attention-heavy workloads, the recommended starting point is D = 128/G - 1, where G is query heads per KV head; larger D values should align G × (1 + D) with 128-token kernel tiles. The remaining guidance covers compute-bound GEMMs, low-latency draft-cost tradeoffs, and choosing among external models, EAGLE-3, MTP, DFlash, DSpark, and suffix or n-gram methods; SPEED-Bench, TensorRT LLM, and NVIDIA/Model-Optimizer support measurement, training, and quantization workflows.


### [Enterprise-Grade Precision for Long-Context Multimodal Embedding Inference on Cloud TPU](https://yomu.fyi/post/enterprise-grade-precision-for-long-context-multimodal-embedding-infer.md)
- Company: [Google](https://yomu.fyi/company/google.md)
- Author: Anthony Su, Injae Kwak
- Published: Aug 26, 2026
- Type: Benchmark

Embedding models convert text, images, and audio into dense vectors used for semantic search, recommendation, intent classification, content discovery, and clustering. The post describes production support for long-context Qwen3 Embedding models on Cloud TPUs through native vLLM TPU integration and GKE Custom Compute Classes, allowing prioritized autoscaling across TPU and GPU capacity pools. Its implementation pads vocabulary tensors for hardware-safe sharding, hardens lazy loading, pre-warms sharding-aware JAX/XLA compilation caches, and uses a hybrid StepPool with CachedRequestState to preserve pooling state during chunked prefill and preemption. Parity evaluations target cosine similarity of at least 0.999 for text and 0.995 for multimodal inputs; on TPU Ironwood, Qwen3-Embedding-8B reached 83,996 total tokens per second and 5.13 requests per second at 16K+ sequence length with bf16 and TP=4.


### [A/B test models in production](https://yomu.fyi/post/a-b-test-models-in-production.md)
- Company: [Together AI](https://yomu.fyi/company/together-ai.md)
- Published: Aug 17, 2026
- Type: Tutorial

A/B testing model changes in production by sending fixed shares of live endpoint traffic to one control and up to 20 variants, allowing teams to compare user-facing outcomes rather than relying only on shadow-traffic checks. The endpoint router keeps variants out of the base traffic split, resamples requests that reach the control, and treats experiment percentages as absolute shares independent of replica counts or autoscaling. A 95/5 experiment can be updated to 80/20 and then 50/50 by resending the complete member list; an etag rejects stale concurrent updates. Measurement joins per-deployment platform metrics with product signals using the deployment ID in response metadata. To finish, a winning variant is promoted through a blue-green rollout before deleting the experiment, while deletion alone returns traffic to the control and can unwind a losing test.


### [DeepSeek-V4 Flash 0731 vs GPT-5.6 Luna on DeepSWE: Cost and Coding](https://yomu.fyi/post/deepseek-v4-flash-0731-vs-gpt-5-6-luna-on-deepswe-cost-and-coding.md)
- Company: [Together AI](https://yomu.fyi/company/together-ai.md)
- Published: Aug 6, 2026
- Type: Benchmark

On the DeepSWE coding benchmark, GPT-5.6 Luna achieves 67.2% pass@1 versus DeepSeek-V4 Flash’s 53.3%, a 14-point gap. DeepSeek costs $0.10 per task, roughly six times less than Luna’s $0.61, delivering 532 solves per $100 versus Luna’s 110. A cascade that runs DeepSeek first and escalates only on failure solves 78.9% of tasks at $0.385 each, beating Luna’s solo accuracy and cutting cost by 37%. DeepSeek fails more gracefully (9% broken tests vs 15%) but is slower and dramatically weaker on JavaScript (35% vs 60%). Luna dominates seven of eight task domains, while DeepSeek leads only on query and config languages, illustrating how inexpensive models can combine with flagships for superior price-performance.


### [Together AI announces strategic partnership with Moonshot AI to natively serve Kimi models](https://yomu.fyi/post/together-ai-announces-strategic-partnership-with-moonshot-ai-to-native.md)
- Company: [Together AI](https://yomu.fyi/company/together-ai.md)
- Published: Jul 29, 2026
- Type: Announcement

Together AI announces a strategic partnership with Moonshot AI to serve Kimi models as a launch platform, beginning with Kimi K3 and extending to future open-weights releases. Kimi K3 is described as a 2.8-trillion-parameter sparse Mixture-of-Experts model with native vision and a 1M-token context window, using Kimi Delta Attention, Attention Residuals, per-head Muon, and quantile-based expert load balancing. Together says developers can access the models through Serverless, Provisioned Throughput, and Dedicated Inference, with US-hosted infrastructure, zero data retention, autoscaling, and a 99% uptime SLA for Provisioned Throughput. The partnership also supports post-training through full-weight and LoRA reinforcement learning and supervised fine-tuning, with checkpoints deployable directly to inference; Kimi K3 is available at launch, while future Moonshot releases are planned for the same platform.


### [ThunderAgent: 2x Faster Agentic Inference for Synthetic Data Generation at Scale](https://yomu.fyi/post/thunderagent-2x-faster-agentic-inference-for-synthetic-data-generation.md)
- Company: [Together AI](https://yomu.fyi/company/together-ai.md)
- Published: Jul 29, 2026
- Type: Announcement

ThunderAgent is a program-level scheduler for high-throughput agentic inference, developed by researchers at Georgia Tech, UIUC, CMU, and Together AI. It solves KV cache thrashing, which occurs when request-level engines like vLLM and SGLang evict an agent’s KV cache during tool calls and recompute it on resume, creating a cascade of evictions. ThunderAgent tracks each workflow as a schedulable program, pausing low-priority workflows under memory pressure and routing resumed programs through a global waiting queue to the node with the most available capacity. In synthetic data generation pipelines, it achieves up to 2.5x single-node throughput and 2.4x speedup on 8 nodes with near-linear scaling, and was accepted as an ICML 2026 Spotlight paper.


### [Kimi K3 by Moonshot now available on Modal](https://yomu.fyi/post/kimi-k3-by-moonshot-now-available-on-modal.md)
- Company: [Modal](https://yomu.fyi/company/modal.md)
- Published: Jul 27, 2026
- Type: Announcement

Moonshot released Kimi K3, a 2.8 trillion parameter multimodal mixture-of-experts transformer with a 1 million token context window and native vision, now on Modal. Modal runs the model at 460 tokens per second on release day through a partnership with Moonshot and vLLM, with token-based pricing on a Shared API or dedicated Auto Endpoint. K3 activates 16 of 896 experts per token and uses Kimi Delta Attention along with Attention Residuals to achieve roughly 2.5 times the scaling efficiency of its predecessor K2. Moonshot employed quantization-aware training from the SFT stage with MXFP4 weights and MXFP8 activations, and contributed a new prefix caching implementation to vLLM to address challenges from KDA. Modal deployed a custom DFlash speculative decoding drafter tuned to K3’s architecture, yielding 360 percent faster interactivity and 88 percent higher throughput on agentic workloads.


### [ZeroEntropy is joining Notion](https://yomu.fyi/post/zeroentropy-is-joining-notion.md)
- Company: [Notion](https://yomu.fyi/company/notion.md)
- Author: Sarah Sachs
- Published: Jul 24, 2026
- Type: Announcement

Notion announces it has acquired ZeroEntropy, whose team will join Notion. In less than two years, ZeroEntropy built high-efficiency training and inference for task-specific models using custom kernels, an inference stack, and model-specialization recipes. As a Notion customer, its reranker made unified search up to 30% faster, cut reranking latency by 85%, maintained constant answer quality, and lowered inference cost. ZeroEntropy has also trained zembed-1 and zerank-2 and published the zELO method; founder and CEO Ghita Houir Alami will lead Notion’s newly formed Model Research team. The acquisition supports Notion’s goal of delivering AI that is fast, affordable, and tailored to specific work for businesses of every size.


### [Together AI and Y Combinator partner to launch the first dedicated GPU cluster for the YC community](https://yomu.fyi/post/together-ai-and-y-combinator-partner-to-launch-the-first-dedicated-gpu.md)
- Company: [Together AI](https://yomu.fyi/company/together-ai.md)
- Published: Jul 20, 2026
- Type: Announcement

Together AI and Y Combinator are partnering to provide YC portfolio companies with a dedicated GPU cluster, addressing compute access as a growing bottleneck for AI-native startups. The service is designed for both inference and training, letting founders reserve, provision, and manage GPUs through Together’s self-service portal rather than making long-term commitments or routing usage through YC. Startups can run short-term sprints at long-term rates, use capacity from single-node workloads to larger deployments, and receive GPUs in minutes, while the cluster is already running at full utilization and companies can plan needs months ahead. Together says its research on attention mechanisms and the Mamba architecture targets faster inference and improved unit economics. The partners plan to expand the cluster and support startups as their requirements grow.


### [What does 99.9% uptime mean for inference?](https://yomu.fyi/post/what-does-99-9-uptime-mean-for-inference.md)
- Company: [Together AI](https://yomu.fyi/company/together-ai.md)
- Published: Jul 16, 2026
- Type: Explainer

The post defines three inference reliability tiers by the failure domains each must survive: 99% requires surviving node-level GPU failures like ECC errors and thermal throttling through automated health checks and fast replica replacement within a single data center. 99.9% requires surviving a full data-center failure by deploying weights across two facilities with enough capacity on each side and routing live traffic to both continuously, not a cold standby. 99.99% requires multi-region deployment with AZ redundancy and reserved failover capacity. The authors argue that vague SLA definitions hide which failure domains are covered, and that infrastructure ownership—whether a provider controls its hardware or rents from a hyperscaler—determines how quickly failures are resolved.


### [Together AI brings Thinking Machines Lab’s new model Inkling on day 0](https://yomu.fyi/post/together-ai-brings-thinking-machines-lab-s-new-model-inkling-on-day-0.md)
- Company: [Together AI](https://yomu.fyi/company/together-ai.md)
- Published: Jul 15, 2026
- Type: Announcement

Thinking Machines Lab released Inkling, and Together AI announced day-zero access through its production inference platform. The multimodal mixture-of-experts model accepts text, image, and audio inputs and generates text through a unified decoder architecture, with controllable inference effort for trading reasoning depth, token usage, and latency. Inkling has 975B total parameters, 40B active parameters per token, a 1M-token context window, query-conditioned relative attention, short causal convolutions, and shared-sink MoE routing; Together serves it with a FlashAttention-4-based kernel. Preliminary evaluations report strong results across scientific reasoning, mathematics, coding, agentic, vision, and audio benchmarks, while the model is also post-trained for forecasting and calibrated prediction. It is available on Together Serverless through OpenAI-compatible APIs without infrastructure provisioning.


### [Open, convenient and predictable: Introducing Provisioned Throughput](https://yomu.fyi/post/open-convenient-and-predictable-introducing-provisioned-throughput.md)
- Company: [Together AI](https://yomu.fyi/company/together-ai.md)
- Published: Jul 8, 2026
- Type: Announcement

Companies adopting open-weight models for production workloads have faced a trade-off between convenient but best-effort serverless inference and powerful but complex dedicated GPU instances. Together AI’s Provisioned Throughput fills that gap with reserved inference capacity sold as Provisioned Throughput Units (PTUs). Each PTU is priced at $0.05 per minute and guarantees a fixed rate of tokens per minute for a specific model, with a 99% uptime SLA. Input, cached input, and output tokens draw down capacity at different rates, letting teams optimize for their traffic shape. Available now for MiniMax M3 and GLM-5.2 with capacity in North America and EMEA and a one-month minimum term, at full utilization costs can run roughly $0.36 per million input tokens and $2.16 per million output tokens, up to 90% below proprietary alternatives.


### [Multi-token Residual Prediction](https://yomu.fyi/post/multi-token-residual-prediction.md)
- Company: [Modal](https://yomu.fyi/company/modal.md)
- Published: Jul 1, 2026
- Type: Explainer

Multi-Token Residual Prediction (MRP) adapts multi-token prediction for diffusion language models by training a small module to predict the residual between adjacent denoising steps rather than the full next-step distribution. This residual target is easier to learn, allowing a tiny 3-layer transformer to accurately predict corrections across multiple steps. In the static regime, MRP enables lossless speculative decoding with up to 1.56× throughput in SGLang, or direct decoding with a small quality cost. In the dynamic regime, MRP recovers up to +16 accuracy points lost by aggressive low-threshold decoding through a remasking mechanism. The module is trained on a frozen backbone and composes cleanly with existing inference methods.


### [Announcing our $800M Series C to accelerate the shift to open-source AI](https://yomu.fyi/post/announcing-our-800m-series-c-to-accelerate-the-shift-to-open-source-ai.md)
- Company: [Together AI](https://yomu.fyi/company/together-ai.md)
- Published: Jul 1, 2026
- Type: Announcement

Together AI announces an $800M Series C to accelerate its open-source AI platform as production demand for inference grows. The company says the round includes $800 million from investors including Aramco Ventures, NVIDIA, Vista Equity, General Catalyst, and others, plus commitments for more than 500 MW of compute capacity to be capitalized independently by new investors. Together argues that closed frontier-model economics become unsustainable as agents and workflows scale, while open-weights models such as DeepSeek, Nemotron, MiniMax, Kimi, and GLM can provide equal or better performance at lower cost. Its full-stack approach spans models, kernels, compilers, inference systems, training infrastructure, hardware utilization, and software, with cited work including FlashAttention-4, Together Megakernel, together.compile, and expanded post-training APIs for tool calling, reasoning, and vision-language models.


### [Together AI at ICML 2026: frontier research across the full stack](https://yomu.fyi/post/together-ai-at-icml-2026-frontier-research-across-the-full-stack.md)
- Company: [Together AI](https://yomu.fyi/company/together-ai.md)
- Published: Jun 30, 2026
- Type: Announcement

Together AI says nine papers from its researchers and peers were accepted to ICML 2026 in Seoul, spanning frontier agents, model shaping, algorithmic and systems optimization, and kernels. The collection is presented as a full-stack view of AI research, with work ranging from DSGym’s unified evaluation and training API for more than 1,000 data-science tasks to ThunderAgent’s program-aware scheduling, TTT-Discover’s test-time reinforcement learning, and RARO’s verifier-free adversarial training. Reported results include up to 3.6× agent throughput, state-of-the-art discoveries from an open 120B model, a 25% expert-response win rate without a verifier, and 5M-token context on one node for Untied Ulysses. Aurora illustrates the research-to-production loop: its adaptive speculative-decoding work ships as the ATLAS speculator on the Together platform.


### [Achieve state-of-the-art inference latencies with speculative decoding](https://yomu.fyi/post/achieve-state-of-the-art-inference-latencies-with-speculative-decoding.md)
- Company: [Modal](https://yomu.fyi/company/modal.md)
- Published: Jun 24, 2026
- Type: Problem & solution

Modal launched Auto Endpoints, combining speculative decoding with Blackwell GPUs, SGLang engine, and Modal Servers to achieve low inference latency for production deployments. The decode phase, where each token requires a full forward pass loading model weights, is the primary latency source, making speculative decoding the most impactful optimization. Speculative decoding runs a separate DFlash speculator model in parallel with the target model using KV projections and a custom Triton kernel, achieving near-linear speedup proportional to acceptance length. In the Decagon voice case study, Modal cut 100ms from a 290ms baseline by mid-training a custom speculator on synthetic data, beating proprietary providers by 60ms. Modal plans to automate speculator training with continuous improvement on production data, moving toward autospec for Auto Endpoints.


[Newer posts](https://yomu.fyi/topic/inference.md) · [Older posts](https://yomu.fyi/topic/inference/page/3.md)