How Caddy Serves a Static Site: The Request Flow
See how Caddy handles a request for a static site, from DNS resolution through container routing to file_server, and how TLS and redirects fit into the flow.
Loading visual explainer…
How Blackwell Serving Turns Peak FLOPs Into Cheap Tokens
Blackwell inference is a pipeline, not a chip upgrade. This explainer walks five stages — fleet validation, batching, KV caching, low-precision kernels, and speculative decoding — that turn peak FLOPs into cheap tokens.
Explanation
A Blackwell GPU only becomes useful inference capacity after several layers of engineering are applied on top of it. The flow starts with validating the GPU fleet against real distributed workloads, then batches many sequences together so matrix math accelerators stay busy instead of waiting on single requests. KV state is cached so shared prefixes are not recomputed, and low-precision formats such as NVFP4 and MXFP8 let the hardware run near peak without changing model behavior. Fused expert kernels and speculative decoding then cut the remaining overhead, which is what turns a ruinously expensive service into a price-competitive one.
FAQ
Why do batching and KV caching matter more than peak FLOPs?
A single request leaves the matrix accelerators mostly idle, so serving economics depend on keeping many sequences in flight at once. Caching KV state avoids recomputing shared prefixes, and Modal reported 2.8x per-user and 5.6x across-user gains after combining batching, KV caching, and NVFP4 micro-scaling on B200 GPUs.
When is speculative decoding worth the extra memory?
A drafter model adds a small memory footprint in exchange for proposing several candidate tokens per step, which the target model then verifies. Liquid AI's 280M-parameter drafter delivered 2.30x to 3.13x faster decoding on an M5 Max without changing output quality, so it pays off when decode latency dominates.
What should be checked before trusting a Blackwell fleet with real workloads?
Standard health checks can pass while a single degraded GPU or misrouted link still fails under real distributed load. NVIDIA's Cluster Readiness Engine runs topology-aware workloads first and attributes each failure to a specific node and category, such as NCCL bandwidth, before production jobs land.
The posts behind it
Related reading
NVIDIA · · TutorialEfficient MoE Training for Biological Foundation Models
Mixture-of-experts (MoE) architectures scale model capacity more efficiently than dense transformers by activating only a subset of experts per token, but naive implementations suffer from fragmented expert computation, routing overhead, and memory pressure. NVIDIA Transformer Engine (TE) addresses these bottlenecks with optimized primitives including GroupedLinear for batched expert GEMMs, MXFP8 low-precision training, and a fused GroupedMLP kernel that combines quantization, SwiGLU activation, and routing-weight scaling. The BioNeMo recipe implements these techniques for biological foundation models, using TE’s autocast API for MXFP8 precision and the Sequential API to fuse the gate-up, ScaledSwiGLU, and down projections into a single operation. On eight NVIDIA B200 GPUs, the recipe achieved up to 2.21x the training throughput of the Hugging Face baseline for a Mixtral-8x7B model. The tutorial provides prerequisites, code examples, and configuration commands to reproduce the optimizations.
Hugging Face · · AnnouncementAccelerating vision-language models with LFM2.5-VL-DSpark
Liquid AI released an experimental DSpark draft model for their vision-language model LFM2.5-VL-3B. The drafter adds a speculative decoding path that trades a minimal increase in memory footprint for a larger speedup without changing output quality. The 280M-parameter drafter, a 4-layer attention-only transformer, captures the target model’s hidden states at fixed tapped layers to draft a block of candidate tokens. On an M5 Max, decoding runs 2.30x to 3.13x faster by task, with end-to-end latency improvements of 1.56x to 2.62x. The drafter ships with day-one support for llama.cpp, MLX-VLM, and SGLang, and is available on Hugging Face in Safetensors and GGUF formats.
- NVIDIA · · Explainer
Validate GPU Cluster Readiness Before AI Workloads Land
GPU clusters can pass standard health checks yet fail under real AI training workloads due to a single degraded GPU, network link, or misrouted traffic. NVIDIA Cluster Readiness Engine (NVCRE) is an open source Kubernetes controller that runs real distributed workloads across topology-aware node groups before production jobs land, measuring results and reporting which nodes failed each test. Its layered API uses Custom Resource Definitions for Certification, Workflow, and Job resources, attributing each failure to a specific node and category such as NCCL bandwidth or training goodput. The WorkloadRun API handles platform detection, NCCL settings, shared-memory volumes, and gang scheduling, so operators do not need to write those manifests by hand. NVCRE integrates with NVSentinel for passive telemetry and is available under Apache 2.0, with a roadmap for inference and automated lifecycle validation.
- NVIDIA · · Problem & solution
Manage Kubernetes Node Fleets with NodeWright
Managing Kubernetes node fleets, especially GPU nodes, is difficult because host-level changes can disrupt long-running training jobs, and traditional tools are not cluster-aware. NodeWright, an open-source Kubernetes-native package manager from NVIDIA, solves this by declaratively configuring and updating node operating systems while respecting PodDisruptionBudgets and non-interruptible workload labels. The operator orchestrates a sequence of cordon, wait, drain, apply, interrupt, and uncordon on each node, using container image packages that carry modifications such as kernel parameters, security agents, and verification scripts. It supports progressive rollouts across a fleet via DeploymentPolicy compartments with configurable disruption budgets and strategies like batch, canary, and rolling. NodeWright integrates with NVIDIA AI Cluster Runtime for validated configurations and is part of the DSX OS open-source portfolio, available under Apache 2.0.
Modal · · ExplainerHow to serve trillions of tokens for trillion-parameter coding agents
Serving inference for coding agents requires trillion-parameter sequence models like Moonshot AI’s Kimi K2.6 to process trillions of input and output tokens at extremely high relative and absolute performance. To be economically viable, these services must achieve large fractions of hardware peak performance, using contemporary matrix math accelerators like Tensor Cores at petaFLOP per second scale. Modal optimized by batching multiple sequences together, caching KV state, and patching SGLang with NVFP4 micro-scaling on Blackwell GPUs like the B200. Per-replica performance improved 2.8x per user and 5.6x across users, transforming ruinously expensive into a price-competitive service. One deployment processed hundreds of billions of tokens daily and trillions total, showing that careful engineering and benchmarking are key to scaling inference.
More like this
More how it works explainers
How BYOK Works: Your Key and Knowledge at Request Time
BYOK lets a system run on assets you supply — your model credential or your private knowledge — instead of the vendor's defaults. This walkthrough traces six steps from registration to an audited response.
APRA CPS 230: Building an AI Stack You Can Swap Out
APRA's April 2026 letter flags single-provider concentration as a CPS 230 risk, so AI stacks need a rehearsed substitution path. This flow maps five stages, from workload classification to provider swap.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
The best engineering reads, once a week
One useful email. Unsubscribe anytime.