Five Checkpoints to Bitwise-Deterministic Training Runs
A five-checkpoint workflow for making large-scale training runs bitwise reproducible: pick the guarantee, fingerprint the run, and trace divergence down to the kernel.
Loading…
Topic
GPU
100 posts about GPU. Every summary links to the original.
- 100 posts
- 15 companies
- 8 explainers
See the idea
GPU explainers
The GPU-Resident Astronomy Pipeline
A GPU-resident astronomy pipeline keeps every stage of image analysis — load, align, subtract, detect, review — inside GPU memory instead of round-tripping through the CPU.
Bare Metal: Performance Bet vs. Operations Tax
Bare metal gives engineers direct control over GPU, network, and virtualization layers, but it shifts elasticity and lifecycle work onto the team. This trade-off map shows when that bargain pays off.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
Manage Kubernetes Node Fleets with NodeWright
Managing Kubernetes node fleets, especially GPU nodes, is difficult because host-level changes can disrupt long-running training jobs, and traditional tools are not cluster-aware. NodeWright, an open-source Kubernetes-native package manager from NVIDIA, solves this by declaratively configuring and updating node operating systems while respecting PodDisruptionBudgets and non-interruptible workload labels. The operator orchestrates a sequence of cordon, wait, drain, apply, interrupt, and uncordon on each node, using container image packages that carry modifications such as kernel parameters, security agents, and verification scripts. It supports progressive rollouts across a fleet via DeploymentPolicy compartments with configurable disruption budgets and strategies like batch, canary, and rolling. NodeWright integrates with NVIDIA AI Cluster Runtime for validated configurations and is part of the DSX OS open-source portfolio, available under Apache 2.0.
How I massively improved my AI inference performance without buying new hardware
Default inference configurations for large language models often leave significant performance unused, even on expensive hardware like 16 NVIDIA H200 GPUs across two nodes. The author tested this setup with vLLM and Red Hat AI Inference 3.4 on a CoreWeave Kubernetes cluster, using enterprise RAG workloads with 2,000-token inputs and 100 concurrent users. By systematically comparing aggregated and prefill-decode (PD) disaggregated architectures, and tuning tensor parallelism (TP) and PD split, latency was dramatically reduced. For Qwen3-32B, time-to-first-token (TTFT) on the best aggregated configuration (2xTP8) dropped from 995ms to 304ms, and with optimized routing to 287ms — a 71% improvement without hardware changes. For Llama-70B, TTFT fell from 1,292ms to 463ms, a 64% gain. The author also built ServeIt Studio, an open-source tool that automates the full optimization pipeline on Kubernetes.
How to serve trillions of tokens for trillion-parameter coding agents
Serving inference for coding agents requires trillion-parameter sequence models like Moonshot AI’s Kimi K2.6 to process trillions of input and output tokens at extremely high relative and absolute performance. To be economically viable, these services must achieve large fractions of hardware peak performance, using contemporary matrix math accelerators like Tensor Cores at petaFLOP per second scale. Modal optimized by batching multiple sequences together, caching KV state, and patching SGLang with NVFP4 micro-scaling on Blackwell GPUs like the B200. Per-replica performance improved 2.8x per user and 5.6x across users, transforming ruinously expensive into a price-competitive service. One deployment processed hundreds of billions of tokens daily and trillions total, showing that careful engineering and benchmarking are key to scaling inference.
Topology-Aware Workload Scheduling with NVIDIA Topograph
AI factories are power-limited systems where GPU workload placement is a key optimization; poor placement fragments topology domains and forces traffic across shared links, reducing throughput and raising job costs. NVIDIA Topograph discovers cluster topology from cloud APIs or on-premises fabric systems, normalizes it into a common model, and publishes it in formats each workload manager expects, such as Kubernetes node labels, Slurm topology configuration, or Slinky ConfigMaps. It uses five components: API Server, Node Observer, Node Data Broker, Provider, and Engine to keep the topology view current as the cluster changes. The post covers deploying Topograph with Helm on Kubernetes, as a system package on Slurm, and as a Helm chart on Slinky, with support for cloud providers including Google Cloud, Lambda, Nebius, Nscale, and OCI. Topograph works alongside Dynamic Resource Allocation and KAI Scheduler to enable topology-aware gang scheduling, improving AI factory efficiency and tokens per watt.
Simplifying Model Serving Across Multiple GPUs with NVIDIA TensorRT Multi-Device Integration in NVIDIA Dynamo-Triton
NVIDIA TensorRT multi-device inference enables a single TensorRT network to execute across multiple GPUs using NCCL-backed distributed collectives, supported from TensorRT 11.0. Integrated into Dynamo-Triton release 26.07, one model instance can own multiple GPUs and expose a single gRPC endpoint so the client does not coordinate GPU ranks. The post demonstrates this with Cosmos 3 Nano video generation, where the transformer uses Ulysses context parallelism to distribute 44,160 video tokens across up to eight GPUs. End-to-end latency drops from 156.6 seconds on one GPU to 34.2 seconds on eight GPUs, achieving a 6.09x transformer RPC speedup. Teams can trade additional GPU resources for shorter request latency while keeping the application interface and surrounding workflow stable.
Benchmarking LLM Inference at Scale with AIPerf
AIPerf is NVIDIA’s successor to GenAI-Perf, intended to benchmark large-scale LLM inference without letting the load client become the bottleneck. Its multiprocess architecture combines worker processes, separate record-processor services, and ZMQ coordination, while supporting 15+ endpoint types, public datasets, and trace-replay formats. The walkthrough serves Qwen3-0.6B through vLLM, using streaming synthetic requests with fixed 128-token inputs and outputs; AIPerf reports TTFT, ITL, request latency, output token throughput, percentile distributions, and optional GPU telemetry in CSV and JSON. A second profile uses Poisson arrivals averaging 10 requests per second, variable input and output lengths, and a fixed random seed, producing wider latency distributions than the static baseline as concurrency and workload variability increase.
How a global fintech scaled coding agent traffic with Dedicated Model Inference
A global fintech runs its coding assistant on GLM 5.2 through Together’s Dedicated Model Inference (DMI), but spiky engineering-hours traffic outpaced static capacity planning. DMI gives the customer’s teams direct control over endpoint provisioning, scaling, configuration, model changes, and performance data through APIs, UI, and CLI. The workload is peak-load and relatively low-TPS, with ISL p50/p90/p95 of roughly 81K/163K/178K tokens and RPS p50/p90/p95 of 3/6/7, so concurrency headroom mattered more than raw throughput; the deployment used dozens of B200s, multiple replicas, and 256K context. When a reshaped endpoint later reached near-100% prefill capacity, requests queued for one to three minutes and decode throughput fell to roughly 5 tokens per second, the team restored cache-session-aware routing and widened max-inflight-per-worker, pushing the change same-day with zero downtime.
Translating CUDA Tile Operations from Python to Rust Using Agentic AI
cuTile Rust (cutile-rs) extends Rust’s ownership model to tile-based GPU kernels, and this post describes an agentic workflow for translating cuTile Python and Triton-TileIR implementations. Because all three front ends target CUDA Tile IR and share the tileiras compiler, the workflow compares reference and translated IR while making Python’s implicit specializations explicit through Rust const generics, fixed dtype symbols, and FFI checks. A bounded multi-agent pipeline covers analysis, device kernels, host and FFI code, correctness, and benchmarking, with machine-checkable verdicts controlling progress. It ported all 24 public TileGym operators, roughly 40 kernels, and reported 99.5% average cuTile Python performance across 347 paired configurations on NVIDIA DGX B200. The skill ships with TileGym and uses acceptance checks requiring correctness, backend execution, and a geomean speedup of at least 0.95.
How NVIDIA Groq 3 LPX Deterministic Execution Drives Power-Efficient High-Interactivity Inference on NVIDIA Vera Rubin
AI factories are constrained by site power, making performance per watt especially important for high-interactivity inference. The NVIDIA Groq 3 LPX rack uses deterministic execution: its compiler schedules computation, data movement, and clock-cycle timing across 256 LPU chips, then derives a cycle-level current-demand curve. Preemptive Power (PEP) prepares the power-delivery network for upcoming spikes, while Clock Period Synthesis (CPS) lengthens or shortens selected cycles to reduce di/dt, voltage droop, and the voltage guardband. Internal testing reported more than 60% less voltage drop, while the source estimates a high-single-digit baseline-voltage reduction and potentially low-double-digit workload-power reduction; paired with Vera Rubin NVL72, the platform is described as delivering up to 35x higher throughput per megawatt than GB200 NVL72 for 2T+ parameter models at long context and high interactivity.
How NVIDIA NVLink 6 Delivers Multi-Layer Resiliency for AI Factories
NVIDIA NVLink 6 is presented as a scale-up networking fabric for large AI factories, where dropped packets, link degradation, and node interruptions can reduce training goodput or inference capacity. It connects 72 Rubin GPUs in the Vera Rubin NVL72 into a single scale-up domain and uses a multi-layer resiliency stack spanning physical, link, application, software, and system layers. Physical Layer techniques include lightweight FEC, Physical Layer Retry (PLR), UPHY recovery, and hardware replay buffers, while credit-based flow control (CBFC) prevents injection when the next hop lacks buffer capacity. Redundant switch trays and management paths, local fault containment, NMX high availability, checkpointing, topology-aware recovery, and targeted switch maintenance are described as mechanisms for containing failures and preserving workloads; the post concludes that this integrated design targets continuous operation and maximum cluster productivity.
Accelerating Dropless MoE Training in JAX with NVIDIA Transformer Engine
Production-scale mixture-of-experts (MoE) training must handle uneven, data-dependent expert loads, creating ragged tensors and expensive dispatch, combine, and all-to-all communication. Dropless MoE processes every selected token, preserving model quality but requiring kernels that support variable token counts without padding or dropping. The post presents JAX and NVIDIA Transformer Engine optimizations centered on group-aware MXFP8 quantization, grouped GEMM, and fused expert-parallel (EP) dispatch and combine using NCCL EP. Grouped GEMM computes each expert’s actual workload in one kernel path, while EP reduces movement overhead and can deduplicate tokens crossing networks. For DeepSeek-V3 training on NVIDIA GB200, the reported throughput increased from 103 to 1,068 TFLOPS/GPU, a 10.4x improvement; the post also reports roughly 10x throughput improvement and 97% scaling efficiency at 1,024 GPUs.
Understanding W8A8 INT8 LLM quantization: Accuracy and performance results
The post evaluates whether W8A8 INT8 quantization can reduce the size of a Llama 3.1 8B Instruct model without sacrificing accuracy or inference performance. Using SmoothQuant to address activation outliers and GPTQ with error compensation for BF16 weights, the model was compressed from 14.9 GB to 8.0 GB after calibration on WikiText-2. Identical zero-shot lm-eval-harness tests across MMLU, ARC Easy, HellaSwag, and IFEval found MMLU, ARC Easy, and HellaSwag deltas smaller than their standard errors, while the IFEval change was described as likely within a similar noise range; the post concludes no measurable degradation. vLLM and GuideLLM tests on one NVIDIA L40S reported 29% more concurrent requests, 24% lower synchronous TTFT, and 44% faster token generation at maximum load, while ITL degraded more quickly under high concurrency because of dynamic activation quantization overhead.
Product updates: Sandbox Sidecars, new models, a refreshed dashboard, and more
Modal added day-zero support for Kimi K3, Qwen 3.8, GLM 5.3, and GLM 5.3 Flash on Auto Endpoints, with custom DFlash speculators boosting throughput to 460 tokens per second. New environment-level budgets give Team and Enterprise workspaces per-environment compute limits, while the pricing multiplier for broad region selection dropped from 1.5x to 1.15x. The dashboard got a refreshed navigation, color system, and beta light mode, and Private Environments now support a default No Access role. Sandbox Sidecars are in public alpha, running additional containers via low-latency internal networking, and VM Sandboxes are in public beta with a real Linux kernel for Docker and eBPF. Sandbox directory snapshots and a new filesystem API are now GA, and regional Proxies allow selecting a region for static outbound IPs, while SDK updates improved log APIs and autoscaler reporting.
Building Pinterest’s VLM Serving Stack on NVIDIA Dynamo
Pinterest describes building a VLM serving stack on NVIDIA Blackwell B200 GPUs and NVIDIA Dynamo for multimodal products including Pinterest Assistant, hybrid search, multimodal reranking, content understanding, signal generation, and content safety. Unlike text-only workloads, requests may contain thousands of images, costly and variable prefill, long multi-turn histories, and large KV caches, making preprocessing, routing, memory, and latency central serving concerns. The implementation combines Dynamo's distributed orchestration, multimodal frontend and vLLM support, KV-aware routing, disaggregated encoder/prefill/decode serving, and LMCache-based KV offloading with vLLM, an OpenAI Chat Completions interface, model-aware routing, EKS, and custom deployment components. Production-oriented AIPerf DAG benchmarks model QPS patterns, multi-turn conversations, many-image requests, and agentic DAGs; the team concludes that payload design, cache-aware routing, offload tiers, and a shared platform are essential to meeting product targets and accelerating new launches.
How Full-Stack NIM Optimizations Deliver 2.5x More Users on Nemotron 3 Ultra
Production serving for agentic AI must maximize concurrent users on fixed GPU infrastructure while preserving latency, especially with long prompts, reused context, and streamed responses. The article presents NVIDIA NIM as a deployable microservice that packages validated model- and GPU-aware choices, then measures Nemotron 3 Ultra NIM 2.0.12 against an open-source baseline on four B200 GPUs using a 64K/400 workload, 76% KV reuse, and 50 TPS per user (20 ms ITL). The optimized stack combines autotuned MoE and Mamba kernels, tensor and expert-aware parallelism, prefix and Mamba state-cache reuse, scheduling, batching, memory tuning, and MTP speculative decoding. At the stated target, throughput rises from 718 to 1,997 output tokens per second—more than 2.5x—while the article recommends replaying representative traffic with NVIDIA AIPerf to select a Pareto point meeting the application’s latency constraint.
Introducing preemptible compute: the same compute, half the price
Together GPU Clusters is introducing preemptible compute in public preview for Kubernetes clusters in all regions, offering interruptible GPU capacity at a flat 50% of the on-demand rate. Preemptible nodes use the same NVIDIA accelerated compute, can be reclaimed when unused capacity is needed, and receive a maximum five-minute drain window for checkpointing and graceful exit. Clusters retain the requested preemptible target and automatically refill as capacity returns, while usage is metered every one to two minutes and billed sub-hourly. The feature suits resumable, retryable, or requeueable work such as experiments, batch inference, and temporary bursts, but critical services, uncheckpointed multi-day runs, and strict-SLO serving should remain on standard nodes; users must schedule eligible pods with the together.ai/compute-class=preemptible label.
To Infinity and Beyond: ThunderKittens Now on NVIDIA Vera Rubin NVL72!
Together’s kernels team ported ThunderKittens to NVIDIA Vera Rubin NVL72 and investigated why existing Blackwell GEMMs underuse the platform. The work targets NVFP4 and FP8 GEMMs: although Rubin preserves Blackwell’s programming model, the original kernels reach only about 42.1% and 44.4% of roofline because tensor cores consume operands twice as fast as the old pipeline supplies them. The team integrates Rubin’s 64-byte K step, exclusive tensor memory extending to 576 columns, oversized shared memory up to 328 KiB, B-side collector buffers, and earlier A-release signaling into its NVFP4 kernel. The resulting kernel exceeds 22 PFLOPS and is described as competitive with cuBLAS and CuTe DSL, while the post also compares Rubin’s specifications with NVIDIA HGX B200.
Anthropic expands Google and Broadcom compute deal
Anthropic announced a new agreement with Google and Broadcom for multiple gigawatts of next-generation TPU capacity expected to come online starting in 2027. The company says the expansion will support its frontier Claude models and meet customer demand, with most of the new compute sited in the United States as an expansion of its November 2025 commitment to invest $50 billion in American computing infrastructure. Anthropic reports that run-rate revenue surpassed $30 billion, up from approximately $9 billion at the end of 2025, while the number of business customers spending more than $1 million annually rose from over 500 to more than 1,000 in less than two months. It will continue using AWS Trainium, Google TPUs, and NVIDIA GPUs, and says Claude remains available across Amazon Web Services, Google Cloud, and Microsoft Azure.
When to Use Encode-Prefill-Decode Disaggregation to Accelerate Multimodal Model Serving
Encode-prefill-decode (EPD) disaggregation separates a multimodal model’s vision encoder from LLM prefill and decode, addressing contention when image or video processing delays language-model work. Using NVIDIA Dynamo, the post compares aggregated serving with colocated encoder workers and a heterogeneous, disaggregated encoder tier, with vision embeddings transferred to PD workers through NIXL over UCX RC/TCP Ethernet. EPD is most useful for media-heavy requests, short-to-medium outputs, smaller or quantized models, and mixed text/multimodal traffic; long decode sequences or light media processing reduce its value. In the reported image-heavy test, colocated and heterogeneous EPD reduced TTFT by 58% and 50%, while the heterogeneous tier served 70% more traffic at the same under-100-ms ITL SLO. Mixed-traffic measurements also reduced mean TTFT for text requests from 92.3 to 53.3 ms.
Higher usage limits and a SpaceX compute deal
An agreement with SpaceX will give the company access to all compute capacity at the Colossus 1 data center, adding more than 300 megawatts and over 220,000 NVIDIA GPUs within the month. Combined with other compute partnerships, the expansion supports higher usage limits: Claude Code five-hour limits double for Pro, Max, Team, and seat-based Enterprise plans, peak-hour reductions end for Pro and Max, and Claude Opus API limits rise considerably. The company says the new capacity will directly improve service for Claude Pro and Claude Max subscribers while it trains and runs Claude on AWS Trainium, Google TPUs, and NVIDIA GPUs. It is also expanding inference capacity in Asia and Europe and exploring orbital AI compute with SpaceX.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.