Topic

Blackwell

1 post about Blackwell. Every summary links to the original.

See the idea

Blackwell explainers

Loading How Blackwell Serving Turns Peak FLOPs Into Cheap Tokens image
How it works Blackwell GPU Inference Mixture of Experts Quantization Speculative Decoding
How Blackwell Serving Turns Peak FLOPs Into Cheap Tokens
Blackwell inference is a pipeline, not a chip upgrade. This explainer walks five stages — fleet validation, batching, KV caching, low-precision kernels, and speculative decoding — that turn peak FLOPs into cheap tokens.

Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.

Newest first

Modal
How to serve trillions of tokens for trillion-parameter coding agents
Serving inference for coding agents requires trillion-parameter sequence models like Moonshot AI’s Kimi K2.6 to process trillions of input and output tokens at extremely high relative and absolute performance. To be economically viable, these services must achieve large fractions of hardware peak performance, using contemporary matrix math accelerators like Tensor Cores at petaFLOP per second scale. Modal optimized by batching multiple sequences together, caching KV state, and patching SGLang with NVFP4 micro-scaling on Blackwell GPUs like the B200. Per-replica performance improved 2.8x per user and 5.6x across users, transforming ruinously expensive into a price-competitive service. One deployment processed hundreds of billions of tokens daily and trillions total, showing that careful engineering and benchmarking are key to scaling inference.

The best engineering reads, once a week.

Topic

1 post about Blackwell. Every summary links to the original.

See the idea

Loading How Blackwell Serving Turns Peak FLOPs Into Cheap Tokens image
How it works Blackwell GPU Inference Mixture of Experts Quantization Speculative Decoding
How Blackwell Serving Turns Peak FLOPs Into Cheap Tokens
Blackwell inference is a pipeline, not a chip upgrade. This explainer walks five stages — fleet validation, batching, KV caching, low-precision kernels, and speculative decoding — that turn peak FLOPs into cheap tokens.

Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.

Newest first

Modal
How to serve trillions of tokens for trillion-parameter coding agents
Serving inference for coding agents requires trillion-parameter sequence models like Moonshot AI’s Kimi K2.6 to process trillions of input and output tokens at extremely high relative and absolute performance. To be economically viable, these services must achieve large fractions of hardware peak performance, using contemporary matrix math accelerators like Tensor Cores at petaFLOP per second scale. Modal optimized by batching multiple sequences together, caching KV state, and patching SGLang with NVFP4 micro-scaling on Blackwell GPUs like the B200. Per-replica performance improved 2.8x per user and 5.6x across users, transforming ruinously expensive into a price-competitive service. One deployment processed hundreds of billions of tokens daily and trillions total, showing that careful engineering and benchmarking are key to scaling inference.

The best engineering reads, once a week.