Loading…
Topic
107 posts about GPU. Every summary links to the original.
See the idea
Apple Silicon's defining choice is one shared memory pool for CPU, GPU, and Neural Engine. This diagram maps that fabric, the on-package bandwidth it buys, and the scaling limits it imposes.
A five-checkpoint workflow for making large-scale training runs bitwise reproducible: pick the guarantee, fingerprint the run, and trace divergence down to the kernel.
A GPU-resident astronomy pipeline keeps every stage of image analysis — load, align, subtract, detect, review — inside GPU memory instead of round-tripping through the CPU.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
JetBrains announced Junie Local is now supported on the NVIDIA RTX Spark platform from day one, and Windows support has moved from experimental to stable. The RTX Spark pairs a Blackwell RTX GPU with a Grace CPU, enabling local AI coding on slim laptops and compact desktops. In testing, Junie Local reached generation speeds of up to 55 tokens per second on the RTX Spark, using Blackwell’s Tensor Cores for FP8 computation and speculative decoding. Junie Local automatically sizes the context window to fit available GPU memory, and users can choose between Qwen3.6-27B or a merged Qwen3.8-3.6-27B-blend model. The local engine now features a simple installer, reliable startup and updates, and no token charges or quotas, with code never leaving the machine.
Databricks’ AI Research team uses AI Runtime (AIR) to manage hundreds of parallel research workstreams for products like Genie, Smart Routing, and AI Functions, training models from a few billion to nearly a trillion parameters. AIR performs preflight checks on GPU nodes to ensure health and communication before training, and monitors node health during execution to detect issues like NCCL failures or thermal throttling. The AIR CLI and custom Docker containers enable fast experimentation by launching jobs in seconds without rebuilding environments, supporting both supervised fine-tuning and reinforcement learning. MLflow integration automatically tracks experiments, while Unity Catalog FUSE mounting gives governed access to research data across workspaces without changing training code. The team has scaled to agent-led training, where AI agents propose changes, launch trials, and evaluate results, increasing training job volume by 8x across AI Functions research workloads.
The post describes systematic performance optimization to reduce fine-tuning time for the Qwen 3.8 27B model on AI Runtime from 7 hours 44 minutes to 1 hour 17 minutes, a 6x improvement. The baseline showed 98% GPU utilization, but torch traces revealed the CPU was stalling the GPU with too many small operations. Installing manually fused kernels such as flash-linear-attention and causal-conv1d reduced the number of GPU operations by 15x and improved training speed by 3.7x. Further gains came from switching from 2D padded batches to 1D packed tensors with correct position_ids, which reduced the number of batches by 36% and eliminated dynamic shape recompilation. The post emphasizes measuring what matters beyond GPU utilization and leveraging pre-built community kernels and efficient batching.
Modal crossed $1B in annualized run rate and raised $750M in Series D funding at a $15.75B post-money valuation, led by Accel. Usage has tripled since the Series C five months ago, with year-over-year growth exceeding 15x, and the company now powers applications from companies including DoorDash, Cognition, and Runway. To avoid the friction of the old stack, Modal built its own file system, container runtime, scheduler, and networking from scratch on a capacity pool spanning over 20 clouds. Sandboxes, launched in 2023 for running LLM-generated code, now represent a $300M run rate product with nearly 3 billion sandboxes spun up and scaling to a million concurrent. New products include multi-node training with a single decorator giving a fully RDMA-connected cluster, Modal servers for HTTP serving, Endpoint Candidates for LLM serving, and generally available virtual machines.
User-defined functions (UDFs) in Apache Spark break the declarative abstraction, forcing GPU-accelerated pipelines to fall back to CPU execution and incur data transfer overhead. NVIDIA cuDF addresses this with GPU parallelism, but UDFs remain a bottleneck due to their opaque code. The post describes three approaches to rewrite UDFs into GPU-accelerable forms: SQL expressions, the RapidsUDF interface using cuDF columnar operations, or custom CUDA kernels via JNI. The cuDF skills for Apache Spark automate this process with four agent-guided feedback loops for testing, conversion, benchmarking, and optimization, providing a project template and best practices. A production deployment converted over 30 UDFs, achieving a 22x speedup on a 600-billion-row workload, reducing operator time from 274.55 to 12.78 hours.
Meta introduced GEM, the industry’s largest recommendation system foundation model, and its 2026 generation shifts from isolated per-domain optimizations on Facebook, Instagram, and Reels to a single central foundation model. The Kunlun architecture, co-designed with the latest GPU hardware, enables 2.5x model complexity and demonstrates LLM-like scaling laws with block-aligned sparse attention reducing sequence complexity. The LoopFM post-training technique transfers GEM’s internal representations to downstream models, doubling intelligence transfer at no additional compute cost or latency. GEM delivered 18.9% conversion improvement and 12% ad click-through rate improvement across all Meta ad surfaces and funnel stages, including 5.8% conversion uplift on Facebook, 4.1% on Instagram, and 1.6% on off-site conversion. Training efficiency doubled to 20-25% model FLOPS utilization while scaling total training FLOPs by 4x across thousands of GPUs using custom kernels and 5D parallelism.
NVIDIA has released the complete Run:ai platform as open source software, expanding the NVIDIA DSX AI Factory Platform and giving customers, developers, and ecosystem partners production-grade AI workload and GPU orchestration capabilities. Run:ai has dynamically allocated GPU resources and managed workloads on Kubernetes across data processing, training, and inference. The release includes the full platform plus four independently adoptable components: KAI Scheduler, GPU Fractioning Operator, KAI Resource Management, and Workload Map. Teams can deploy the stack from the NVIDIA Run:ai repository in the DSX AI Factory GitHub organization or via an NGC Community Product Helm chart, without a paid subscription or enterprise support entitlement. Ecosystem partners including Crusoe, Mirantis, Nebius, OpenNebula Systems, Rafay, Saturn Cloud, Spectro Cloud, and vCluster are integrating the platform or its components into their offerings.
TinyFish builds infrastructure for enterprises that need agents to work across the live web and is developing its own specialized web-native model. The company uses Databricks AI Runtime to manage GPU compute for post-training with supervised fine-tuning and moving toward Direct Preference Optimization, MLflow for evaluation, and Model Serving for production deployment. TinyFish evaluates its model by measuring full agent task completion rather than individual responses, achieving a 77.6% success rate on its internal go/no-go benchmark where 1,691 of 2,178 tasks were completed successfully. Over a 15-day period, the web-native model handled 294,781 agent runs across 6.4 million tool steps at an estimated serving cost of $0.088 per run or $0.0040 per tool step. By consolidating training, evaluation, serving, and operational data on Databricks, TinyFish reduces infrastructure overhead and maintains a continuous development cycle from agent behavior to deployment.
Minions is a dynamic, distributed virtual GPU cloud that aggregates idle Apple Silicon GPUs across macOS devices to run large-scale machine learning inference. The work responds to limits in centralized GPU infrastructure, including high cost, limited elasticity, and long wait times when resources are constrained. Demand for GPU-accelerated machine learning inference keeps rising, while modern Apple Silicon macOS devices carry powerful GPUs that stay largely idle during normal usage. Minions offers flexible GPU inference task execution and provides fault tolerance and efficient resource utilization, letting teams use existing hardware in place of additional GPU procurement. The authors report that idle macOS devices can become a practical distributed GPU inference platform in environments where centralized GPU resources are constrained.
Ai2 replaced its priority-based GPU cluster scheduler with a system combining GPU time budgets, hierarchical fair-share allocation, and a time-slicing contract to prioritize high-impact research while maintaining full occupancy. The previous priority scheduler induced GPU squatting, priority inflation, and constant negotiation over non-preemptable workloads, exemplifying a tragedy of the commons. In the new model, managers allocate proportional GPU time budgets to projects, making resource allocation a transparent administrative process rather than an operational debate. The scheduler tracks occupancy over a time window with weights set by budgets, and time-slicing enforces minimum runtime protection before workloads become preemptible. However, interactive sessions suffered from the 8-hour cap, prompting investment in a CPU-only cluster and restorable sessions, while potential capacity fragmentation for large workloads remains under investigation.
AI factories combine GPUs, CPUs, switches, DPUs, and SuperNICs with schedulers and security controls, making validation before hardware arrival critical. NVIDIA DSX Air provides a node-based digital twin that models hardware topology and software stack, allowing teams to validate supported configurations in a safe environment before production. AI agents interrogate the twin, run supported configuration and security checks, compare results with organizational policy, and produce evidence-based reports. The simulation supports Day 0 through Day 2 operations and integrates with NVIDIA Brev for on-demand GPU compute to run AI services like Video Search and Summarization. This governed automation model uses human-in-the-loop gates to ensure agents explore in the sandbox while production changes require approval, enabling continuous improvement.
Supply chain and energy grid planning problems now exceed the memory and time limits of single-GPU solvers. NVIDIA cuOpt introduces mPDLP, a multi-GPU primal-dual hybrid gradient solver that distributes linear programming problems across NVLink-connected GPUs. The algorithm’s hot loop is a sparse matrix-vector multiplication, a memory-bound operation that benefits from the increased bandwidth of multiple GPUs. Min-cut partitioning reduces communication overhead by keeping tightly connected dependencies on the same GPU, enabling up to 6x lower peak memory per GPU. On the benchmark problem zib03 with over 104 million nonzeros, mPDLP achieved a solve rate nearly 10x faster than the previous year, and partners Kinaxis and PSR demonstrated 3.3x and 5x speedups on real-world models.
NVIDIA cuPhoton is an open-source CUDA-X toolkit of GPU-accelerated building blocks for scientific image analysis, from sensor read through classification, covering spectral and optical astronomy and time-domain laser/X-ray analysis. High-throughput instruments like the Vera C. Rubin Observatory generate up to 20 TB of images per night, but CPU-bound pipelines create delays of hours to months. cuPhoton keeps data on the GPU end-to-end, collapsing nine months of data analytics to four hours on GPU-accelerated Python. The toolkit includes modules for FITS loading, image registration, PSF-matching and subtraction, dipole fitting, and candidate review. Benchmarks on multi-GPU systems show individual operation speedups of up to 14,900x for loading and 14,550x for processing.
Bitwise determinism ensures independent and checkpoint-resumed training runs follow the same numerical trajectory, providing four key benefits: replaying failures, preserving checkpoint continuity, detecting unexpected differences, and comparing changes without run-to-run variation. At trillion-parameter scale across thousands of GPUs, nondeterminism complicates failure reproduction and fix validation. NVIDIA Megatron Core targets two guarantees: independent reproducibility and bitwise checkpoint resume. The post describes a validation procedure using full-precision metrics and a progressive tracing workflow that uses torch.hash_tensor for GPU-resident fingerprints to narrow divergence from end-to-end metrics to the responsible kernel. A Nemotron case study demonstrates that kernel optimizations, such as separating grouped-GEMM writers and applying a fixed reduction order, can reduce the determinism performance overhead to approximately 2% steady-state determinism tax at scale.
NVIDIA DOCA GPUNetIO is a GPU-centric networking SDK that enables CUDA kernels to drive Ethernet, RDMA, Verbs, and DMA operations with the CPU removed from the application critical path. It unifies previously fragmented GDA-KI-style RDMA implementations from libraries such as NCCL, NVSHMEM, UCX, and the Aerial 5G SDK into a single shared foundation. The framework provides both a full DOCA SDK version and a lighter open-source Verbs-focused implementation; the open-source version can optionally dlopen the SDK at runtime. On the data path, CUDA kernels directly post Work Queue Entries, ring the NIC doorbell, and poll for completions without CPU intervention. Benchmarks using NVSHMEM show that the GPU-initiated path avoids the CPU proxy bottleneck, achieving higher bandwidth for small message sizes compared to CPU-mediated communication.
GPU-accelerated Kubernetes clusters depend on compatible versions across host kernels, GPU drivers, container runtimes, networking, storage, operators, and workload frameworks, each on its own release cycle. A configuration that works for one service, GPU generation, and Kubernetes release may silently fail for another, and tracing version conflicts after deployment is slow. NVIDIA AI Cluster Runtime (AICR) v1.0 addresses this with version-locked, validated recipes that pin component combinations, render deployment artifacts for Helm, Argo CD, Flux, or Helmfile, and carry signed validation evidence from tested hardware. The release establishes a stable compatibility contract across its CLI, REST API, Go SDK, bundle layout, and artifact schemas, with merge-blocking checks on public interfaces. AICR now has over 100 contributors, with almost half from outside NVIDIA, and integrations from Pulumi Labs and Mirantis demonstrate consuming the same configuration through different tools.
Green contexts, available in the CUDA Driver API since 12.4 and the Runtime API since 13.1, let applications explicitly partition GPU execution resources such as SMs and workqueues. They are lightweight to create and destroy, do not implicitly synchronize unrelated work, and allow provisioning workqueues to reduce false dependencies. Streams are created from a green context rather than relying on implicit device state, making targeting explicit. Benchmarks on a Blackwell GPU show a critical kernel completing in 0.007 ms with green context partitioning, compared to 0.140 ms with high-priority streams and 3.727 ms with equal priority. The tradeoff is fewer SMs for the bulk workload, but green contexts are opt-in and can be adopted incrementally alongside existing CUDA code.
Together AI is collaborating with IBM and NVIDIA to scale enterprise-grade AI inference using a dedicated cluster of NVIDIA B300 GPUs on IBM Cloud with NVIDIA Spectrum-X Ethernet networking. This is the first large-scale inference cluster of its kind on IBM Cloud, and Together AI is the first customer running on it. Together AI operates the inference layer, IBM provides the cloud, and NVIDIA delivers the silicon and networking. The company serves hundreds of trillions of tokens per month to over a million developers, and demand continues to climb. Enterprises choose open models to keep their data sovereign and achieve frontier-level performance at a fraction of closed-model cost.
The Kubernetes device plugin model treats GPUs as anonymous integers, making fair sharing impossible because a full 80 GB GPU and a 5 GB partition both consume ‘1’ from the same quota. Dynamic resource allocation (DRA), generally available in Kubernetes 1.34, replaces this by having drivers advertise real device attributes like memory and compute capability, but DRA alone does not address quota management or preemption. Red Hat build of Kueue 1.4 on Red Hat OpenShift integrates with DRA to enforce quota, borrowing, and preemption for both whole GPUs and partitionable devices using device-count or counter-based memory accounting. For partitionable GPUs, Kueue reads counter data from DRA drivers and charges GPU memory against ClusterQueue quota instead of device count, enabling accurate fair sharing between training and inference workloads. Future developments include extended resources for simpler adoption without ResourceClaimTemplates, consumable capacity for dynamic time-slicing, and scheduler-library integration to verify scheduling feasibility.
Olmo-core 3 is an open training infrastructure upgrade for large mixture-of-experts (MoE) models, designed to scale to trillion parameters while preserving efficiency. It switches from FSDP to a DDP-based system that keeps experts resident on GPUs and routes data to them, avoiding weight gathering overhead. The system combines expert, pipeline, and distributed optimizer parallelism with optimizations like rowwise expert parallelism, GPU-resident routing, and grouped GEMM. In a benchmark, increasing experts from 8 to 128 while routing each token to four experts kept active parameters at ~3.2B, total parameters grew from 4.6B to 47B with throughput falling less than 5%. The infrastructure supports MXFP8 lower-precision format, achieving about 21% higher throughput than BF16, and has been tested on a 1.2-trillion-parameter model.
The best engineering reads, once a week.
