Five Checkpoints to Bitwise-Deterministic Training Runs
A five-checkpoint workflow for making large-scale training runs bitwise reproducible: pick the guarantee, fingerprint the run, and trace divergence down to the kernel.
Loading…
Topic
GPU
100 posts about GPU. Every summary links to the original.
- 100 posts
- 15 companies
- 8 explainers
See the idea
GPU explainers
The GPU-Resident Astronomy Pipeline
A GPU-resident astronomy pipeline keeps every stage of image analysis — load, align, subtract, detect, review — inside GPU memory instead of round-tripping through the CPU.
Bare Metal: Performance Bet vs. Operations Tax
Bare metal gives engineers direct control over GPU, network, and virtualization layers, but it shifts elasticity and lifecycle work onto the team. This trade-off map shows when that bargain pays off.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
Personalize your product's text-to-speech voice for any language: Fine-tuning with Kubeflow Trainer on Red Hat OpenShift AI
Training an English-centered text-to-speech model for Turkish can produce fluent but hard-to-understand pronunciation for native speakers. This tutorial fine-tunes unsloth/orpheus-3b-0.1-pretrained with about 81,000 Turkish text-audio pairs on Red Hat OpenShift AI using a Kubeflow Trainer TrainJob distributed across two nodes with two A100 GPUs each. LoRA freezes the 3-billion-parameter base and trains 97.3M adapter parameters, while preprocessing converts text to Llama-3 tokenizer tokens and audio to three-layer SNAC codes, masking text labels during scoring. The merged model is evaluated against the base on ten Turkish sentences using Whisper transcription and WER/CER metrics. Mean WER falls from 2.44 to 0.35 and CER from 1.59 to 0.09, though the authors describe the small test as directional.
Introducing CUDA Rust: Two Tracks for Writing GPU Kernels
NVIDIA’s CUDA Rust effort targets the GPU-kernel gap in Rust-based systems, where host code can be Rust but kernels often require another language. It presents two tracks: cuda-oxide compiles #[kernel] functions through Rust MIR, Pliron, and LLVM IR to PTX for SIMT programming, while cutile-rs expresses computations as tiles and JIT-compiles embedded ASTs through CUDA Tile IR. The SIMT example uses DisjointSlice, typed thread indices, and launch contracts to constrain per-thread output access and validate launch configuration; Tile leaves thread mapping and shared-memory management to its compiler. The post recommends starting with Tile because the compiler decides tile mapping per architecture, and using SIMT when explicit thread or memory control is needed. Both projects are early-stage, with cuda-oxide in early alpha and cutile-rs further along but still incomplete and subject to API changes.
Understanding W8A8 INT8 LLM quantization: Half the size, better performance, same accuracy
Serving Llama 3.1 8B in BF16 requires roughly 14.9 GB for weights, leaving little VRAM on a 16 GB GPU for KV cache and activations, which limits practical deployment. The workflow benchmarks the base model, applies INT8 W8A8 quantization with llm-compressor, then repeats accuracy and performance tests under identical hardware and load conditions. SmoothQuant modifies weights to reduce activation outliers, while GPTQ uses calibration data and error compensation before rounding weights to INT8; the experiment used 512 WikiText-2 samples and targeted deployment with vLLM. On the Llama 3.1 8B Instruct model, compression reduced storage from 14.9 GB to 8.0 GB, a 46% reduction, and accuracy was preserved across MMLU, ARC, HellaSwag, and IFeval; generation-speed results are reserved for part 2.
Achieving Extreme Efficiency through Specialized GPU Kernel Generation
Production inference systems often use generic GPU kernels even though operation shapes combine static model parameters with request-time dimensions such as token counts. Proteus addresses this mismatch by having agents propose specialized kernels, then applying controlled-reference validation, static checks, isolated repeated timing, hidden tests, and consistency checks before retaining measured winners. Its knowledge layer uses actionable, scoped lessons retrieved through hierarchical tags and hybrid keyword-semantic search, while background jobs handle deeper distillation instead of burdening each attempt. On NVIDIA B200 GPUs with a Triton backend, a Gated DeltaNet packed-decode case for Qwen 3.5 122B produced shape-specific wins, including 1.6× for a Batch=4, Key=128, Value=128 path and 0.018 ms latency for Candidate 030. Across the reported kernels, Proteus achieved 1.8–5.2× speedups over the best available in vLLM, with validation and context management identified as the main difficulties.
Speeding up LLM inference with P-EAGLE in vLLM Speculators
P-EAGLE (Parallel EAGLE), implemented in Speculators v0.6.0, extends EAGLE-3 with parallel drafting so a draft model predicts multiple future tokens in one forward pass rather than sequentially. Speculative decoding pairs this small draft model with a verifier model, which checks the proposed sequence in parallel and accepts only its correct prefix while preserving the verifier’s output distribution. The post adds Conditional Drop-token (COD) sampling, learnable mask embeddings and hidden states, and a custom flex-attention mask to support multi-depth prediction within the EAGLE-3 framework. The post presents an end-to-end workflow covering data preparation, vLLM hidden-state extraction, training and deployment, including a Qwen3-8B example that takes about 50 minutes on four H100s. A published Qwen3-a8B draft model reports an average length of 3.044 across the listed datasets.
The Modern CUDA Toolbox in Practice: A Step-by-Step Optimization Walkthrough
An image-processing pipeline serves as the test case for a modern CUDA toolbox, starting with RGB-to-grayscale conversion, 32-by-32 tile median computation, and host/device transfers. It advances through six incremental changes: adopting CCCL and Compute Sanitizer for indexing bugs, adding NVTX for Nsight Systems benchmarks, replacing hand-rolled work with CUB algorithms, using pooled device containers and pinned host containers, and assigning each image its own stream with asynchronous copies. The implementation uses cuda::stream, cuda::copy_bytes, cuda::device_buffer, and CUB calls on per-thread streams, allowing kernels and transfers to overlap instead of executing sequentially on the default stream. The reported total duration falls from 6.8 seconds to about 23 milliseconds for three images, a roughly 300x improvement, while the post characterizes the code as safer and easier to maintain without low-level optimizations.
Co-Designing AI Models Using Speculative Decoding for Faster LLM Inference
Speculative decoding accelerates large language model inference by having a smaller draft model propose multiple tokens and a target model verify them in parallel, preserving the standard output sequence under its acceptance criteria. It reduces decoding iterations and increases target-model arithmetic intensity, but speedup depends on acceptance length, verification time, draft latency, batch size, and hardware behavior. For attention-heavy workloads, the recommended starting point is D = 128/G - 1, where G is query heads per KV head; larger D values should align G × (1 + D) with 128-token kernel tiles. The remaining guidance covers compute-bound GEMMs, low-latency draft-cost tradeoffs, and choosing among external models, EAGLE-3, MTP, DFlash, DSpark, and suffix or n-gram methods; SPEED-Bench, TensorRT LLM, and NVIDIA/Model-Optimizer support measurement, training, and quantization workflows.
LLM quantization guide: How to do it, and how it helps
LLM quantization addresses the gap between rapidly growing model sizes and GPU memory by storing weights and activations at lower precision than BF16. It explains how quantization targets linear layers in transformer blocks, distinguishes weight quantization for memory-bound decoding from activation quantization for compute-bound prefill, and maps formats such as GGUF, FP8, NVFP4, AWQ-INT4, and MLX to their inference engines. Calibrated methods including GPTQ, AWQ, and SmoothQuant protect important weights using representative data, while reported NVFP4 results recover about 99% of BF16 accuracy for 70B–235B models and 95–98% for 7B–14B models. In a Llama 3 70B benchmark on two H100s, FP8 weights-and-activations increased throughput from 158 to 474 input tokens/sec and reduced time to first token under load from over 30,000ms to about 4,800ms.
How Botika runs full-stack generative AI on Modal
Botika builds agentic e-commerce teams that automate visual production for global fashion brands, using proprietary foundation models and a 100-terabyte image dataset. The company runs its entire full-stack AI operation on Modal, with the data pipeline processing 100 terabytes at less than 1% error rate across thousands of containers. Before Modal, the team struggled with Kubernetes fleet management and slow GCP Batch jobs; now a single researcher can launch 50 experiments daily using Claude, and they set up reinforcement learning infrastructure in less than a day. Modal handles autoscaling, cold starts, and multi-node training with RDMA, allowing Botika to double traffic in seconds, and they serve about 15 models on L4s, L40S, A100s, and H100s. The infrastructure is not Modal-specific, enabling the team to run open-source and custom frameworks without modification.
How to Size GPUs for AI Inference and TCO Without Overspending
GPU sizing for AI inference depends on use case, token patterns, latency targets, concurrency, cache hit rate, model choice and deployment strategy rather than raw hardware specifications alone. The post proposes a core-and-flex capacity model, combining baseline on-prem or reserved cloud capacity with elastic spot or on-demand GPUs for surges, launches and experiments. It also recommends matching GPU memory to prompt lengths and concurrency, while using quantization, pruning and distillation to reduce footprint and operating cost. A NeMo example prunes Qwen3-8B from 36 to 24 layers or narrows its dimensions toward a roughly 6B student before teacher-student distillation. In the illustrative run, width pruning reached lower final validation loss than depth pruning, 3.21 versus 3.60, while depth pruning converged faster.
Run NVIDIA BioNeMo NIM Microservices for Protein Structure Prediction in Claude Science
Claude Science is used with the NVIDIA BioNeMo Agent Toolkit and NVIDIA NIM microservices to investigate how the predicted Seh1 structure changes alone versus with proposed partner C1HCX1. The workflow retrieves UniProt sequences, generates unpaired and species-paired multiple-sequence alignments with MSA Search, and sends those inputs to OpenFold3 and Boltz-2 for independent monomer and heteromer predictions. Using MSA input, heteromer iPTM reached 0.85 for OpenFold3 and 0.82 for Boltz-2, versus 0.14 and 0.19 without it; five-sample runs had standard deviations no greater than 0.006. Both models placed partner β-strands at Seh1’s WD40 propeller closure, while core Cα-RMSD remained 0.68 Å and 0.65 Å, respectively, but the predicted interaction remains experimentally unverified.
Fine-Tuning SOTA Object Detection Models on Real-World Datasets
This practical comparison examines how YOLO12, YOLO26, and RF-DETR behave when moved from COCO to specialized real-world object-detection datasets, and how fine-tuning changes the outcome. The authors first evaluate six pretrained checkpoints on COCO val2017, then use cable-damage, bone-fracture, and soda-bottles datasets from RF100-VL for zero-shot testing, fine-tuning, and accuracy/latency comparison. Because the model implementations require incompatible dependencies, the workflow uses one PyCharm project with three isolated uv environments and remote GPU execution; the reported latency stays in native frameworks rather than TensorRT. All models perform well on COCO, but direct performance on specialized data falls close to zero, while 10 fine-tuning epochs adapt the detectors well to cable damage and soda bottles. Bone-fracture remains difficult despite fine-tuning, with RF-DETR performing best, indicating that domain shift may require more data, longer training, or domain-specific pretraining.
Stop wasting GPU allocation in Kubernetes with GPU-pruner
GPU-pruner addresses idle GPU allocation in Kubernetes by identifying workloads whose hardware activity remains below a configured threshold, rather than relying on browser sessions, network traffic, or pod lifetime. It queries Prometheus for NVIDIA Data Center GPU Manager metrics, including engine activity and GPU utilization, and treats peak performance below 1% throughout a default 35-minute observation window as idle. After tracing an idle pod through ownerReferences or KServe labels, it scales the parent resource to zero, preserving workload metadata for later restoration. Slack notifications, grace periods, lookback checks, and dry-run mode provide safeguards, while Prometheus label handling must preserve the workload namespace and pod identity in DCGM telemetry.
Claude Science, an AI workbench for scientists
Claude Science is a beta AI workbench for scientists that brings databases, research tools, computing resources, and domain-specific workflows into one environment. Researchers can analyze literature, run multi-step studies, create figures and manuscripts, and revise them through conversation. Every generated artifact includes its code, environment, plain-language creation description, and message history, while a reviewer agent checks citations, calculations, and consistency between figures and code. A coordinating agent uses more than 60 curated skills and connectors across fields including genomics, single-cell, proteomics, structural biology, and cheminformatics; it can manage jobs on local systems, HPC clusters over SSH, or Modal, scaling from one GPU to hundreds. Beta access is available on macOS and Linux for Pro, Max, Team, and Enterprise users, with reported examples including faster glioma analyses and large literature reviews.
Mixture-of-Kittens: our open-source MoE megakernel for NVL72s
Mixture-of-Kittens (MoK) is an open-source production mixture-of-experts (MoE) training megakernel designed for GB300 NVL72 systems, where the MoE layer had become a major training bottleneck and could consume more than half of end-to-end training time. It redesigns the layer around a single fully deterministic kernel that fuses communication and computation, minimizing CPU work and CPU-GPU synchronization while overlapping operations across the 72-GPU NVLink domain. The post describes target-specific scheduling, push-versus-pull communication choices, pipelined expert computation, ring token buffers, megakernel structure, determinism, and MXFP8 support for DeepSeek-V3-style layers. Across four model shapes, MoK is reported as up to 2.37x faster for MXFP8 forward and 1.78x for backward, while production training on 512 GPUs reached 1.41x tokens per second versus a DeepEP-based implementation. MoK is fully open-sourced and intended to remain modifiable for other platforms.
GEM Training: How Meta Doubled the Efficiency of Its LLM-Scale Ads Foundation Model
Meta’s Generative Ads Recommendation Model (GEM), a hybrid foundation model for ads recommendations on Instagram and Facebook, trains across several thousand latest-generation GPUs and combines trillions of sparse embedding parameters with billions of dense parameters. The post frames its engineering problem around recommendation-specific jagged and asymmetric sequences, memory-bound operations, numerical sensitivity, communication overhead, activation pressure, and load skew that make conventional LLM training techniques insufficient. Its co-design approach combines custom kernels such as Jagged Flash Attention (JFA), BlockAttention, and Generalized Dot-Product Attention (GDPA), mixed ultra-low-precision training including MXFP8 attention and MLP, topology-aware 5D parallelism, SM-free collectives, and memory and load-balancing techniques. The reported result is a doubling of end-to-end training efficiency to 20–25% Model FLOPs Utilization (MFU) while total training FLOPs increased fourfold over 12 months; Base Batch Shuffling (BBS) alone delivered a reported 4% efficiency gain, with 4% QPS improvement and 4% peak memory reduction.
Autoscaling endpoints for LLM inference
Dedicated LLM inference deployments face a trade-off between paying for idle GPUs and suffering sharp p95 latency when replicas hit batching limits. The autoscaling loop converts observed load into desired replicas with ceil(N × observed/target), then applies scale-up and scale-down windows, replica bounds, and GPU placement. It presents inflight_requests as a leading, engine-agnostic default, while TTFT and end-to-end latency target responsiveness but trail demand, and GPU or token utilization prioritize efficiency with less latency headroom. Cold starts can take several minutes because replicas require GPU placement, weight downloads, VRAM loading, and warm-up, so policies should scale early; streaming is required for TTFT, decoding speed, and throughput-per-replica metrics, whereas end-to-end latency supports both streaming and non-streaming workloads.
Together AI and Y Combinator partner to launch the first dedicated GPU cluster for the YC community
Together AI and Y Combinator are partnering to provide YC portfolio companies with a dedicated GPU cluster, addressing compute access as a growing bottleneck for AI-native startups. The service is designed for both inference and training, letting founders reserve, provision, and manage GPUs through Together’s self-service portal rather than making long-term commitments or routing usage through YC. Startups can run short-term sprints at long-term rates, use capacity from single-node workloads to larger deployments, and receive GPUs in minutes, while the cluster is already running at full utilization and companies can plan needs months ahead. Together says its research on attention mechanisms and the Mamba architecture targets faster inference and improved unit economics. The partners plan to expand the cluster and support startups as their requirements grow.
What does 99.9% uptime mean for inference?
The post defines three inference reliability tiers by the failure domains each must survive: 99% requires surviving node-level GPU failures like ECC errors and thermal throttling through automated health checks and fast replica replacement within a single data center. 99.9% requires surviving a full data-center failure by deploying weights across two facilities with enough capacity on each side and routing live traffic to both continuously, not a cold standby. 99.99% requires multi-region deployment with AZ redundancy and reserved failover capacity. The authors argue that vague SLA definitions hide which failure domains are covered, and that infrastructure ownership—whether a provider controls its hardware or rents from a hyperscaler—determines how quickly failures are resolved.
New in Together GPU Clusters: Reliability and control for production GPU clusters
Together GPU Clusters introduces changes for production training and inference workloads, addressing hardware failures, scheduler problems, and the limits of single-admin access. Platform health improvements include continuously running passive health checks, human-approved node repair recommendations, and Slurm-on-Kubernetes 2.0, which adds self-healing daemons, durable job accounting, descendant-process cleanup, and refreshed GPU state after reschedules. Operational controls include a cluster details view with health, utilization, events, configuration, and node actions, plus external OIDC, startup scripts, and an acceptance-test opt-out. Newly provisioned Slurm clusters use the latest stack by default, while existing managed clusters can be migrated during a maintenance window at no cost; the stated result is faster recovery, fewer support tickets, and greater visibility and control for growing teams running large clusters.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.