---
title: "Inference"
description: "51 posts about Inference, summarised, each linking to the original."
---

# Inference
> 51 posts about Inference, summarised, each linking to the original.

## Articles

### [Introducing Modal Auto Endpoints: Optimized inference you actually own](https://yomu.fyi/post/introducing-modal-auto-endpoints-optimized-inference-you-actually-own.md)
- Company: [Modal](https://yomu.fyi/company/modal.md)
- Published: Jun 23, 2026
- Type: Announcement

Modal Auto Endpoints is a self-serve service for production-grade LLM inference that gives teams ownership of the entire serving stack, not just an API. Unlike proprietary providers, it exposes the inference code, engine flags, and GPU selection, and provides detailed metrics like speculative decoding acceptance length and per-replica token latency quantiles in a dashboard. Built on Modal’s infrastructure, it uses autoscaling to handle traffic spikes by spinning up additional replicas automatically, global GPU capacity without capacity management, and Modal Servers for ultra-low-latency routing with only 5ms overhead. For each supported model, it offers a pre-configured deployment using speculative decoding with the DFlash block-diffusion drafter from Z Lab, and teams can benchmark latency and throughput tradeoffs with a click after deployment. The platform is designed with a declarative interface based on workloads and SLOs, with future automation levels including autoinference, autospec, autodistill, and autoresearch for continuous improvement of inference performance.


### [Speculation Is All You Need](https://yomu.fyi/post/speculation-is-all-you-need.md)
- Company: [Modal](https://yomu.fyi/company/modal.md)
- Published: Jun 19, 2026
- Type: Explainer

Speculative decoding is presented as the most important optimization for LLM inference, delivering speedups of 2x-3x rather than small percentage gains from kernel tuning. The technique accelerates the decode phase by using a lightweight draft model to generate candidate tokens that the target model processes in parallel, with lossless resampling. The authors released DFlash speculators for multiple Qwen models, achieving an additional 5-20% speedup over existing baselines, and worked with SGLang to integrate them. On a B200 node, the Qwen 3.5 122B-A10B model with speculation reaches over 1000 tok/s at concurrency 1, compared to 250 tok/s without speculation, demonstrating a 4x speedup. Training these speculators is simplified because the data-generating process is the target model itself, making data collection trivial, and the post envisions a flywheel where custom speculators and distillation enable continuous improvement in latency, cost, and quality for self-hosted inference.


### [Making FlashAttention-4 faster for inference](https://yomu.fyi/post/making-flashattention-4-faster-for-inference.md)
- Company: [Modal](https://yomu.fyi/company/modal.md)
- Author: Charles Frye
- Published: Jun 11, 2026
- Type: Explainer

FlashAttention-4, originally designed for pre-training, required modifications to efficiently handle LLM inference workloads dominated by the memory bandwidth-limited decode phase. Modal’s contributions include adjusting parallelism strategies from query parallelism to key/value parallelism, implementing split KV to parallelize across KV tiles, and adding a heuristic for optimal splits. They added support for FP8 inputs, arbitrary KV page sizes via cp.async loads, and reduced wasted work for small query sizes by using a single query tile. The kernel also gained extended GQA packing for irregular head ratios, using cp.async for query tiles to bypass TMA alignment restrictions. These changes, measured in throughput improvements up to 3x for single-token decode and up to 2.4x for small page sizes, demonstrate the leverage of flexible parallelism and tile-based programming for inference optimization.


### [Modal's Series C: Raising $355M at a $4.65B valuation](https://yomu.fyi/post/modal-s-series-c-raising-355m-at-a-4-65b-valuation.md)
- Company: [Modal](https://yomu.fyi/company/modal.md)
- Published: May 21, 2026
- Type: Announcement

Modal announced a $355 million Series C at a $4.65 billion post-money valuation, led by General Catalyst and Redpoint, with participation from Menlo, Bain Capital Ventures, Accel, and all existing investors. The company grew fivefold since September and surpassed $300 million in annualized revenue. Modal is a cloud platform built for AI workloads, offering elastic inference, dynamic agent runtimes, and batch processing. Its Sandboxes feature, which provides isolated environments for untrusted code, has launched over 1 billion instances and accounts for more than a third of revenue. Modal plans to invest in low-latency inference, reinforcement learning infrastructure, and expanded agent computing capabilities.


### [Blazing fast on-device GenAI with LiteRT-LM](https://yomu.fyi/post/blazing-fast-on-device-genai-with-litert-lm.md)
- Company: [Google](https://yomu.fyi/company/google.md)
- Author: Tenghui Zhu, Yu-hui Chen, Ram Iyengar
- Published: May 19, 2026
- Type: Announcement

LiteRT-LM is presented as an on-device runtime for deploying Gemma 4 across Android, iOS, and the web despite constrained memory, limited compute, and varied hardware. It combines LiteRT inference, quantization, XNNPACK and MLDrift kernels, CPU/GPU/NPU backends, transfer-avoiding pipelines, Multi-Token Prediction (MTP), and session management. Reported decode rates reach 52 tokens/sec on Android GPU, 56 on iOS Metal, and up to 76 on WebGPU; MTP provides up to a 2.2x speedup, while a ~2.58GB E2B model uses 607MB of physical memory on Apple mobile CPUs. The runtime also supports save/restore of KV caches, native vision and audio loading, Thinking Mode, constrained decoding, and function calling. New Swift and JavaScript APIs extend the integration surface, including WASM/WebGPU browser inference designed for serverless, secure, privacy-preserving applications.


### [How we achieved truly serverless GPUs](https://yomu.fyi/post/how-we-achieved-truly-serverless-gpus.md)
- Company: [Modal](https://yomu.fyi/company/modal.md)
- Author: Charles Frye
- Published: May 12, 2026
- Type: Problem & solution

GPU inference workloads suffer from spiky demand, leading to low GPU Allocation Utilization when capacity is fixed. Modal reduced replica startup latency from over 2 kiloseconds to about 50 seconds using four techniques. A buffer of idle GPUs removes instance allocation from the hot path. A custom content-addressed filesystem serves container images lazily from a multi-tier cache. CPU and CUDA checkpoint/restore fast-forward through initialization by directly restoring processes and GPU contexts into memory. These optimizations enable truly serverless GPU inference, as demonstrated by Reducto’s document processing scaling to thousands of GPUs with cold starts reduced from ~70s to ~12s.


### [Boosting multimodal inference performance by >10% with a single Python dictionary](https://yomu.fyi/post/boosting-multimodal-inference-performance-by-10-with-a-single-python-d.md)
- Company: [Modal](https://yomu.fyi/company/modal.md)
- Published: May 4, 2026
- Type: Problem & solution

Multimodal vision-language models pose new challenges for inference engines like SGLang. Profiling with py-spy revealed that the scheduler’s process\_input\_requests consumed about 13% of CPU time, with a quarter of that spent on repeated calls to torch.UntypedStorage.\_new\_shared\_cuda for CUDA IPC handle book-keeping. The solution was a simple Python dictionary cache for pool handles, eliminating redundant opens. On a Qwen2.5-VL-3B-Instruct benchmark with a single H100, throughput increased 16.2% from 22.2 to 25.7 requests per second, mean time-to-first-token dropped 13.2%, and mean time-per-output-token dropped 17.2%. The improvement is merged in SGLang v0.5.10 and applies to any multimodal model using SGLang’s CUDA IPC transport.


### [Early team](https://yomu.fyi/post/early-team.md)
- Company: [Cursor](https://yomu.fyi/company/cursor.md)
- Author: Sualeh Asif
- Published: Apr 26, 2025
- Type: Announcement

Cursor introduces a representative sample of the people on its team and collaborators, spanning infrastructure, distributed systems, AI research, design, performance, and product development. The profiles include Aman Karmani, Jeremy Stribling, Jacob Jackson, Ian Huang, Sasha Rush, Alex Haugland, Ryo Lu, Lukas Möller, Jack Gallagher, and Michael Feldstein, with prior work at GitHub, Keybase, OpenAI, Cornell, Google, Notion, DeepJudge, Midjourney, Figma, and Meta. Their cited work includes KBFS, speculative decoding, a browser-based multiplayer Minecraft engine, language-model efficiency and long-context research, Cursor’s in-house inference engine, and Midjourney pretraining and reinforcement learning. The post invites people interested in transformational AI products, frontier RL, AI infrastructure, or distributed systems to contact hiring@cursor.com.


### [A new Tab model](https://yomu.fyi/post/a-new-tab-model.md)
- Company: [Cursor](https://yomu.fyi/company/cursor.md)
- Author: Phillip Kravtsov & Jacob Jackson
- Published: Jan 13, 2025
- Type: Announcement

Cursor announces Fusion, a next-generation model for Cursor Tab, which predicts edits near the cursor and “jumps” to likely next editing locations. The post positions this as a response to the limits of cursor-only text insertion, with a broader goal of reducing code-editing tedium and moving toward in-flow Next Action Prediction. Compared with the March 2024 model, Fusion predicts over 25% more difficult edits per line, suggests over 10x longer change stretches, cuts median server latency from 475ms to 260ms, expands context from 5,500 to 13,000 tokens, and adds instant, accurate cursor jumps. These gains are attributed to cleaner and larger datasets, longer editor-state prompts, training for larger edits, synthetic instruction-following data, and inference and performance improvements. Fusion is rolling out to all users in client release 0.45.0.


### [We're Cutting L40S Prices In Half](https://yomu.fyi/post/we-re-cutting-l40s-prices-in-half.md)
- Company: [Fly.io](https://yomu.fyi/company/fly-io.md)
- Published: Aug 15, 2024
- Type: Announcement

Fly.io lowered the hourly price of NVIDIA L40S GPUs to $1.25, making them available at the same price as A10s after customer data showed that A10s—not the more powerful cards—were its most popular GPU. The post argues that most users adding GPUs to fast networks are running inference rather than training, making networking, memory, object storage, and request latency more important than access to an H100. The L40S is described as an AI-optimized L40 with AI compute performance comparable to an A100, while the L40 retains graphics hardware and is designed for rack use with more memory and lower energy consumption than gaming GPUs. The intended combination places model parameters, datasets, and compute near one another on Fly.io’s Anycast network and Tigris object storage. Examples include Llama 3.1 70B, Flux, Whisper, SegAlign, and DOOM Eternal.


### [Efficient Visual Representation Learning And Evaluation](https://yomu.fyi/post/efficient-visual-representation-learning-and-evaluation.md)
- Company: [Etsy](https://yomu.fyi/company/etsy.md)
- Author: Eden Dolev
- Published: Jun 7, 2024
- Type: Explainer

Etsy uses image vector representations for visual search and similar-item recommendations, making low-latency, memory-aware models important for online query inference and downstream retrieval. The work compares EfficientNetB0, a compact CNN, with EfficientFormer-l3, which downsamples multiple blocks and applies attention only in its final stage; its representation uses the last block’s eight-head attention and average-pools the sequence. Fine-tuning adds an embedding and multiple classification heads, while multitask training supervises taxonomy, color, and review-photo labels; both models train on two 16GB P100 GPUs. Evaluation rebuilds a brute-force candidate index after each epoch and measures Recall@5 and Recall@10 on interaction and intra-item retrieval datasets, with early stopping available. EfficientFormer-l3 improves reported offline recall by about 5%, 17%, and 1.8% across three tasks, while experiments report lifts in CTR, purchase rate, click-recall@100, and clicks.


[Newer posts](https://yomu.fyi/topic/inference/page/2.md)