What Is an AI Factory? Power In, Tokens Out
An AI factory is a fixed power and cooling budget converted into model output. This explainer maps its four parts and the throughput-per-watt trade-offs that connect them.
Loading…
Topic
Inference
46 posts about Inference. Every summary links to the original.
- 46 posts
- 12 companies
- 3 explainers
See the idea
Inference explainers
How Blackwell Serving Turns Peak FLOPs Into Cheap Tokens
Blackwell inference is a pipeline, not a chip upgrade. This explainer walks five stages — fleet validation, batching, KV caching, low-precision kernels, and speculative decoding — that turn peak FLOPs into cheap tokens.
How BioNeMo Turns a Target List Into Folded Structures
BioNeMo Inference Runtime runs one pipeline per target: parse, tokenize, generate features, infer on a GPU replica, and write a structure. This diagram traces those stages and the throughput trade-off.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
How we achieved truly serverless GPUs
GPU inference workloads suffer from spiky demand, leading to low GPU Allocation Utilization when capacity is fixed. Modal reduced replica startup latency from over 2 kiloseconds to about 50 seconds using four techniques. A buffer of idle GPUs removes instance allocation from the hot path. A custom content-addressed filesystem serves container images lazily from a multi-tier cache. CPU and CUDA checkpoint/restore fast-forward through initialization by directly restoring processes and GPU contexts into memory. These optimizations enable truly serverless GPU inference, as demonstrated by Reducto’s document processing scaling to thousands of GPUs with cold starts reduced from ~70s to ~12s.
Boosting multimodal inference performance by >10% with a single Python dictionary
Multimodal vision-language models pose new challenges for inference engines like SGLang. Profiling with py-spy revealed that the scheduler’s process_input_requests consumed about 13% of CPU time, with a quarter of that spent on repeated calls to torch.UntypedStorage._new_shared_cuda for CUDA IPC handle book-keeping. The solution was a simple Python dictionary cache for pool handles, eliminating redundant opens. On a Qwen2.5-VL-3B-Instruct benchmark with a single H100, throughput increased 16.2% from 22.2 to 25.7 requests per second, mean time-to-first-token dropped 13.2%, and mean time-per-output-token dropped 17.2%. The improvement is merged in SGLang v0.5.10 and applies to any multimodal model using SGLang’s CUDA IPC transport.
Early team
Cursor introduces a representative sample of the people on its team and collaborators, spanning infrastructure, distributed systems, AI research, design, performance, and product development. The profiles include Aman Karmani, Jeremy Stribling, Jacob Jackson, Ian Huang, Sasha Rush, Alex Haugland, Ryo Lu, Lukas Möller, Jack Gallagher, and Michael Feldstein, with prior work at GitHub, Keybase, OpenAI, Cornell, Google, Notion, DeepJudge, Midjourney, Figma, and Meta. Their cited work includes KBFS, speculative decoding, a browser-based multiplayer Minecraft engine, language-model efficiency and long-context research, Cursor’s in-house inference engine, and Midjourney pretraining and reinforcement learning. The post invites people interested in transformational AI products, frontier RL, AI infrastructure, or distributed systems to contact [email protected].
A new Tab model
Cursor announces Fusion, a next-generation model for Cursor Tab, which predicts edits near the cursor and “jumps” to likely next editing locations. The post positions this as a response to the limits of cursor-only text insertion, with a broader goal of reducing code-editing tedium and moving toward in-flow Next Action Prediction. Compared with the March 2024 model, Fusion predicts over 25% more difficult edits per line, suggests over 10x longer change stretches, cuts median server latency from 475ms to 260ms, expands context from 5,500 to 13,000 tokens, and adds instant, accurate cursor jumps. These gains are attributed to cleaner and larger datasets, longer editor-state prompts, training for larger edits, synthetic instruction-following data, and inference and performance improvements. Fusion is rolling out to all users in client release 0.45.0.
We're Cutting L40S Prices In Half
Fly.io lowered the hourly price of NVIDIA L40S GPUs to $1.25, making them available at the same price as A10s after customer data showed that A10s—not the more powerful cards—were its most popular GPU. The post argues that most users adding GPUs to fast networks are running inference rather than training, making networking, memory, object storage, and request latency more important than access to an H100. The L40S is described as an AI-optimized L40 with AI compute performance comparable to an A100, while the L40 retains graphics hardware and is designed for rack use with more memory and lower energy consumption than gaming GPUs. The intended combination places model parameters, datasets, and compute near one another on Fly.io’s Anycast network and Tigris object storage. Examples include Llama 3.1 70B, Flux, Whisper, SegAlign, and DOOM Eternal.
Efficient Visual Representation Learning And Evaluation
Etsy uses image vector representations for visual search and similar-item recommendations, making low-latency, memory-aware models important for online query inference and downstream retrieval. The work compares EfficientNetB0, a compact CNN, with EfficientFormer-l3, which downsamples multiple blocks and applies attention only in its final stage; its representation uses the last block’s eight-head attention and average-pools the sequence. Fine-tuning adds an embedding and multiple classification heads, while multitask training supervises taxonomy, color, and review-photo labels; both models train on two 16GB P100 GPUs. Evaluation rebuilds a brute-force candidate index after each epoch and measures Recall@5 and Recall@10 on interaction and intra-item retrieval datasets, with early stopping available. EfficientFormer-l3 improves reported offline recall by about 5%, 17%, and 1.8% across three tasks, while experiments report lifts in CTR, purchase rate, click-recall@100, and clicks.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.