Five Checkpoints to Bitwise-Deterministic Training Runs
A five-checkpoint workflow for making large-scale training runs bitwise reproducible: pick the guarantee, fingerprint the run, and trace divergence down to the kernel.
Loading…
Topic
GPU
98 posts about GPU. Every summary links to the original.
- 98 posts
- 14 companies
- 8 explainers
See the idea
GPU explainers
The GPU-Resident Astronomy Pipeline
A GPU-resident astronomy pipeline keeps every stage of image analysis — load, align, subtract, detect, review — inside GPU memory instead of round-tripping through the CPU.
Bare Metal: Performance Bet vs. Operations Tax
Bare metal gives engineers direct control over GPU, network, and virtualization layers, but it shifts elasticity and lifecycle work onto the team. This trade-off map shows when that bargain pays off.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
Announcing our $800M Series C to accelerate the shift to open-source AI
Together AI announces an $800M Series C to accelerate its open-source AI platform as production demand for inference grows. The company says the round includes $800 million from investors including Aramco Ventures, NVIDIA, Vista Equity, General Catalyst, and others, plus commitments for more than 500 MW of compute capacity to be capitalized independently by new investors. Together argues that closed frontier-model economics become unsustainable as agents and workflows scale, while open-weights models such as DeepSeek, Nemotron, MiniMax, Kimi, and GLM can provide equal or better performance at lower cost. Its full-stack approach spans models, kernels, compilers, inference systems, training infrastructure, hardware utilization, and software, with cited work including FlashAttention-4, Together Megakernel, together.compile, and expanded post-training APIs for tool calling, reasoning, and vision-language models.
Anthropic integration with Modal brings scalable compute to Claude Science
Anthropic launched Claude Science, an AI workbench where life sciences researchers run computational workloads from conversation with Claude. The workbench addresses the fragmented toolchain of local machines, HPC clusters, and cloud VMs by centralizing execution in one place. Modal integrates with Claude Science as the compute backend, routing GPU and CPU-heavy workloads to Modal sandboxes automatically and fanning out parallel tasks like virtual screening across hundreds of containers to reduce hours to minutes. Researchers can prompt for tasks such as predicting GFP structure, and Claude writes the folding code, runs it on a Modal GPU, and returns the structure with confidence scores inside the conversation, with Modal handling scheduling, shared storage, and reproducible environments. Modal is committing up to $100,000 in compute for the Claude Science Cohort, with 500– 2,000 allocations per project.
Together AI at ICML 2026: frontier research across the full stack
Together AI says nine papers from its researchers and peers were accepted to ICML 2026 in Seoul, spanning frontier agents, model shaping, algorithmic and systems optimization, and kernels. The collection is presented as a full-stack view of AI research, with work ranging from DSGym’s unified evaluation and training API for more than 1,000 data-science tasks to ThunderAgent’s program-aware scheduling, TTT-Discover’s test-time reinforcement learning, and RARO’s verifier-free adversarial training. Reported results include up to 3.6× agent throughput, state-of-the-art discoveries from an open 120B model, a 25% expert-response win rate without a verifier, and 5M-token context on one node for Untied Ulysses. Aurora illustrates the research-to-production loop: its adaptive speculative-decoding work ships as the ATLAS speculator on the Together platform.
How Databricks is turning video into searchable, actionable intelligence
Databricks presents video analysis as a data engineering problem for organizations with terabytes of footage that is difficult and expensive to review manually. An app accepts a video and a natural-language prompt, then triggers a Lakeflow job on Serverless GPU Compute to run Meta’s SAM3 segmentation model frame by frame and retain matching moments. Those clips preserve original timestamps and are sent through the Databricks Foundation Model API for summaries that can be written to tables or passed into downstream workflows. In one example, 26 minutes of traffic footage became one minute and 55 seconds of relevant video. The model-agnostic pipeline uses MLflow signatures to support interchangeable or custom models, while event-driven execution and independent serverless GPUs allow concurrent processing without cluster management or idle GPU costs.
Achieve state-of-the-art inference latencies with speculative decoding
Modal launched Auto Endpoints, combining speculative decoding with Blackwell GPUs, SGLang engine, and Modal Servers to achieve low inference latency for production deployments. The decode phase, where each token requires a full forward pass loading model weights, is the primary latency source, making speculative decoding the most impactful optimization. Speculative decoding runs a separate DFlash speculator model in parallel with the target model using KV projections and a custom Triton kernel, achieving near-linear speedup proportional to acceptance length. In the Decagon voice case study, Modal cut 100ms from a 290ms baseline by mid-training a custom speculator on synthetic data, beating proprietary providers by 60ms. Modal plans to automate speculator training with continuous improvement on production data, moving toward autospec for Auto Endpoints.
Genesis Workbench: A blueprint for industry AI in life sciences, powered by Databricks and NVIDIA
Genesis Workbench is an open blueprint for a Databricks-native life-sciences application that brings computational drug-discovery stages into one governed workbench. It combines Unity Catalog governance, MLflow tracking, Model Serving, serverless GPU compute, Databricks AI Search, and NVIDIA technologies including CUDA-X libraries, Parabricks, BioNeMo tools, GenMol, and Proteina-Complexa. Independent modules cover genomics, single-cell analysis, large- and small-molecule workflows, and model fine-tuning, with handoffs spanning gene-to-sequence resolution, structure prediction, docking, ADMET, and candidate ranking. A point-and-click React interface supports bench scientists, while declarative workflow generation and MCP exposure let pipelines and external clients use the workbench; inference runs on GPU endpoints inside the governed workspace without runtime external API dependencies. The stated aim is to let teams move from disease hypotheses to ranked therapeutic candidates on their own data, with a roadmap for automated workflow generation, BioNeMo Skills integration, and additional MCP services.
ParallelKernelBench: Frontier LLMs can't write fast multi-GPU kernels (yet)
ParallelKernelBench (PKB) measures whether language models can replace PyTorch + NCCL implementations with fast CUDA kernels for multi-GPU workloads, where communication can account for over 20% of inference latency. Its 87 problems come from real codebases and span tensor, context, data, expert, sequence, and FSDP/ZeRO parallelism, with hardware topology supplied and symmetric memory used for direct GPU communication over NVLink. In zero-shot evaluation, GPT-5.5 solved 28 problems and produced 22 faster-than-baseline solutions; sampling three attempts raised those figures to 36 and 27, but the best fast 1@3 result was only 31%. Models performed best on familiar collective primitives, tensor-parallel GEMMs, and Ulysses-style context parallelism, while failures often involved rank coordination, data partitioning, collective ordering, and underuse of TMA or NVLS.
Databricks and NVIDIA: Building for the Agentic Era
Databricks and NVIDIA describe a full-stack partnership for enterprise AI, spanning governed model training, inference, agent infrastructure, and industry-specific AI. Databricks AI Runtime brings NVIDIA Hopper GPUs and Quantum InfiniBand to governed data for distributed training, with Blackwell preparation, Free Edition GPU support, and planned NGC container support. Model Serving uses NVIDIA hardware and Triton Inference Server for low-latency, high-throughput production inference, while NVIDIA Vera targets agent harnesses, tool calls, analytics, and multi-step orchestration with up to 3x faster SQL queries and 80% faster agentic performance. The collaboration also places NVIDIA Agent Toolkit on Databricks Apps, integrates GPU debugging through Genie Code, and brings frameworks such as BioNeMo, Parabricks, cuOpt, and Isaac Sim to Databricks workflows. The stated vision is an end-to-end stack in which GPUs handle models and purpose-built CPUs handle surrounding agentic workloads, with governance supplied by the Databricks platform.
What’s coming next to Free Edition
Databricks is expanding Free Edition, launched twelve months ago to help people learn modern data and AI, after providing more than $10M in credits to over 500,000 learners. The five additions are Genie Code, serverless GPUs subject to availability, Lakebase, Agent Bricks, and Lakeflow Designer. Genie Code can generate, run, interpret, and refine code autonomously for tasks such as dataset analysis, pipeline cleaning, and visualization, while GPUs support deep-learning workloads including training, fine-tuning, inference, and large-dataset processing. Lakebase adds a fully managed Postgres-compatible database for data apps and AI agents; Agent Bricks supplies composable tools, memory, orchestration, and evaluation; and Lakeflow Designer enables visual pipeline construction. Together, these additions are presented as a complete free toolkit covering data engineering, analytics, machine learning, app development, and AI agents, allowing users to build end-to-end data and AI projects in Free Edition.
What’s New in the AI Platform: Agents for ML Engineering, Our Deep Learning Platform, and New Capabilities for Real-Time ML
The announcement presents three additions to the Databricks AI Platform: Genie Code support for ML engineering, AI Runtime’s serverless GPU environment, and expanded real-time ML capabilities. Genie Code integrates with Unity Catalog, Feature Store, training, serving, monitoring, and MLflow, assisting with feature engineering, model training, deployment, evaluation, and production operations. AI Runtime provides on-demand serverless NVIDIA A10 and H100 GPUs, supports high-performance multinode training with RDMA and high-performance data loading, and adds Lakeflow Jobs, DABs, MLflow, and Unity Catalog integration. For real-time ML, the platform adds declarative feature engineering, streaming features, online feature serving on Lakebase, and enhanced Model Serving targeting 300K+ QPS with under 10ms p99 latency overhead. Reported customer examples include faster workflows, lower infrastructure costs, and production scaling beyond 100K QPS.
Making FlashAttention-4 faster for inference
FlashAttention-4, originally designed for pre-training, required modifications to efficiently handle LLM inference workloads dominated by the memory bandwidth-limited decode phase. Modal’s contributions include adjusting parallelism strategies from query parallelism to key/value parallelism, implementing split KV to parallelize across KV tiles, and adding a heuristic for optimal splits. They added support for FP8 inputs, arbitrary KV page sizes via cp.async loads, and reduced wasted work for small query sizes by using a single query tile. The kernel also gained extended GQA packing for irregular head ratios, using cp.async for query tiles to bypass TMA alignment restrictions. These changes, measured in throughput improvements up to 3x for single-token decode and up to 2.4x for small page sizes, demonstrate the leverage of flexible parallelism and tile-based programming for inference optimization.
DiffusionGemma: The Developer Guide
DiffusionGemma is an experimental text-generation model and developer guide covering serving, customization, and non-autoregressive generation. Built on the Gemma 4 backbone, its 26B Mixture of Experts architecture activates 3.8B parameters during inference, refines 256-token canvases in parallel, and shifts the bottleneck from memory bandwidth to compute. Block autoregressive diffusion commits denoised 256-token blocks to the KV cache, combining bidirectional denoising with causal prefill for longer sequences; vLLM integration provides an OpenAI-compatible local server. A Sudoku fine-tuning example using Hackable Diffusion reports improvement from roughly 0% base-model success to 80%, while SFT reduces the demonstrated solve from 48 to 12 steps through adaptive stopping. The guide also identifies bidirectional attention and re-noising as mechanisms for global constraint handling and self-correction.
Introducing the Google Colab CLI
Google announces the Google Colab Command-Line Interface (CLI), a tool that connects a local terminal with remote Colab runtimes for developers and AI agents. It can provision GPUs or TPUs with commands such as colab --gpu A100, execute local Python scripts and ML pipelines remotely with colab exec, recover artifacts and replayable .ipynb logs, and provide interactive access through colab repl or colab console. The source demonstrates an agent-driven QLoRA fine-tuning workflow for google/gemma-3-1b-it on a Text-to-SQL dataset: an Antigravity agent provisions a T4, installs transformers, datasets, peft, trl, bitsandbytes, and accelerate, runs finetune_run.py, saves a notebook log, downloads model artifacts, and stops the runtime. The resulting safetensors adapter and configuration files can be loaded locally, while the CLI is presented as programmable and usable by terminal-based agents.
Anthropic raises $65B in Series H funding at $965B post-money valuation
Anthropic says it has raised $65 billion in Series H funding at a $965 billion post-money valuation, with the round led by Altimeter Capital, Dragoneer, Greenoaks, and Sequoia Capital. The company says its run-rate revenue crossed $47 billion earlier this month and that the financing will support safety and interpretability research, additional compute, and expanded products and partnerships. It reports agreements for up to five gigawatts of new Amazon capacity, five gigawatts of next-generation TPU capacity with Google and Broadcom, and GPU access in SpaceX’s Colossus 1 and Colossus 2; AWS remains its primary cloud provider and training partner. The round includes $15 billion in previously committed hyperscaler investments, including $5 billion from Amazon, alongside strategic infrastructure partners Micron, Samsung, and SK hynix.
Reinforcement learning is an infrastructure problem
Reinforcement learning for post-training large language models has become infrastructure-constrained as teams move to multi-node configurations for larger models like Qwen 3 30B and Kimi K2.6. The RL loop divides into three independent infrastructure challenges: training with reliable forward and backward passes, high-performance inference rollouts, and thousands of concurrent sandbox environments for policy execution. Weight synchronization between trainer and rollout engine becomes the critical bottleneck in multi-node setups, with RDMA offering up to 100x speedup over TCP and significant cost savings per step. Teams commonly get stuck maintaining glue code, queueing for cluster time, and under-utilizing GPUs when sandbox buffers are not correctly sized. Modal addresses these with RDMA-connected clusters via a single rdma=True keyword, fast container boot, and the new open-source Modal Training Gym library that abstracts common scaffolding into under 100 lines of code.
Reliable LLM Inference at Scale
Databricks describes an inference platform serving open-source and proprietary frontier models at more than 125T tokens per month, with reliability as the central challenge under spiky, multi-tenant demand. It identifies failures and latency risks from expensive GPU configurations, variable input and output lengths, long-context requests, multimodal preprocessing, and engine-level hangs. Its architecture combines rate limiting, capacity management, model units, Dicer load-aware sticky routing, and autoscaling; model units estimate multidimensional request cost and allocate capacity more predictably. For runtime reliability, prioritized black-box health checks trigger Kubernetes liveness restarts for silent hangs, completing detection and recovery in less than five minutes while eliminating false probe failures. Image-processing changes, including Torchvision processors and OMP_NUM_THREADS configuration, removed CPU throttling and increased completed requests per second by more than threefold with the same replicas and load; bursty workloads also achieved over 80% GPU savings versus static peak provisioning.
Modal's Series C: Raising $355M at a $4.65B valuation
Modal announced a $355 million Series C at a $4.65 billion post-money valuation, led by General Catalyst and Redpoint, with participation from Menlo, Bain Capital Ventures, Accel, and all existing investors. The company grew fivefold since September and surpassed $300 million in annualized revenue. Modal is a cloud platform built for AI workloads, offering elastic inference, dynamic agent runtimes, and batch processing. Its Sandboxes feature, which provides isolated environments for untrusted code, has launched over 1 billion instances and accounts for more than a third of revenue. Modal plans to invest in low-latency inference, reinforcement learning infrastructure, and expanded agent computing capabilities.
Scaling reinforcement learning at Applied Compute
Applied Compute, founded by former OpenAI Codex and o1 researchers, builds custom AI agents for enterprises through a post-training approach they call Specific Intelligence. Their core training mechanism is Reinforcement Learning, where agents repeatedly attempt tasks in replayable environments and are scored against reward functions to refine model weights. For clients like DoorDash and Cognition, this means training specialized agents for merchant onboarding and bug-catching that improve from continuous use. The company evaluated multiple infrastructure platforms before selecting Modal, which provides ephemeral sandboxes for complex environment simulation, sub-second cold starts to keep GPUs busy, and serverless fan-out for parallel grading. Modal’s primitives allow each phase of the RL loop—rollouts, evals, and inference—to operate with appropriate resources while sharing state efficiently.
How we achieved truly serverless GPUs
GPU inference workloads suffer from spiky demand, leading to low GPU Allocation Utilization when capacity is fixed. Modal reduced replica startup latency from over 2 kiloseconds to about 50 seconds using four techniques. A buffer of idle GPUs removes instance allocation from the hot path. A custom content-addressed filesystem serves container images lazily from a multi-tier cache. CPU and CUDA checkpoint/restore fast-forward through initialization by directly restoring processes and GPU contexts into memory. These optimizations enable truly serverless GPU inference, as demonstrated by Reducto’s document processing scaling to thousands of GPUs with cold starts reduced from ~70s to ~12s.
How Superhuman and Databricks built a 200K QPS inference platform together
Superhuman replaced a DIY vLLM-based serving stack for its grammar-correction model with Databricks model serving, targeting peak traffic above 200,000 QPS, sub-second P99 latency, four-nines reliability, and no quality regression. The platform combines an Endpoint Discovery Service with power-of-two-choices load balancing, concurrency-based autoscaling, and asymmetric scale policies to reduce hotspots and handle rapid traffic ramps. Lazy-loading container images changed startup from several minutes to a few seconds by fetching metadata first and retrieving 4MB blocks on demand. Runtime work on H100 GPUs raised per-pod throughput from 750 to 1,200 QPS: FP8 weight quantization contributed up to 30%, while a multiprocessing server added 20%, alongside C++ and asynchronous scheduling optimizations. The teams validated changes through shadow testing and evaluation harnesses, and Superhuman is migrating additional models and workflows to the platform.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.