---
title: "CUDA"
description: "17 posts about CUDA, summarised, each linking to the original."
---

# CUDA
> 17 posts about CUDA, summarised, each linking to the original.

## Articles

### [Faster Scientific Image Analysis with NVIDIA cuPhoton](https://yomu.fyi/post/faster-scientific-image-analysis-with-nvidia-cuphoton.md)
- Company: [NVIDIA](https://yomu.fyi/company/nvidia-developer-blog.md)
- Author: Quynh L. Nguyen
- Published: Oct 7, 2026
- Type: Announcement

NVIDIA cuPhoton is an open-source CUDA-X toolkit of GPU-accelerated building blocks for scientific image analysis, from sensor read through classification, covering spectral and optical astronomy and time-domain laser/X-ray analysis. High-throughput instruments like the Vera C. Rubin Observatory generate up to 20 TB of images per night, but CPU-bound pipelines create delays of hours to months. cuPhoton keeps data on the GPU end-to-end, collapsing nine months of data analytics to four hours on GPU-accelerated Python. The toolkit includes modules for FITS loading, image registration, PSF-matching and subtraction, dipole fitting, and candidate review. Benchmarks on multi-GPU systems show individual operation speedups of up to 14,900x for loading and 14,550x for processing.


### [How DOCA GPUNetIO Unifies GPU-Initiated Networking Across the NVIDIA Software Stack](https://yomu.fyi/post/how-doca-gpunetio-unifies-gpu-initiated-networking-across-the-nvidia-s.md)
- Company: [NVIDIA](https://yomu.fyi/company/nvidia-developer-blog.md)
- Author: Elizabeth Goodman
- Published: Oct 6, 2026
- Type: Explainer

NVIDIA DOCA GPUNetIO is a GPU-centric networking SDK that enables CUDA kernels to drive Ethernet, RDMA, Verbs, and DMA operations with the CPU removed from the application critical path. It unifies previously fragmented GDA-KI-style RDMA implementations from libraries such as NCCL, NVSHMEM, UCX, and the Aerial 5G SDK into a single shared foundation. The framework provides both a full DOCA SDK version and a lighter open-source Verbs-focused implementation; the open-source version can optionally dlopen the SDK at runtime. On the data path, CUDA kernels directly post Work Queue Entries, ring the NIC doorbell, and poll for completions without CPU intervention. Benchmarks using NVSHMEM show that the GPU-initiated path avoids the CPU proxy bottleneck, achieving higher bandwidth for small message sizes compared to CPU-mediated communication.


### [Control How Your GPU Shares Work with Green Contexts](https://yomu.fyi/post/control-how-your-gpu-shares-work-with-green-contexts.md)
- Company: [NVIDIA](https://yomu.fyi/company/nvidia-developer-blog.md)
- Author: Elizabeth Goodman
- Published: Oct 6, 2026
- Type: Explainer

Green contexts, available in the CUDA Driver API since 12.4 and the Runtime API since 13.1, let applications explicitly partition GPU execution resources such as SMs and workqueues. They are lightweight to create and destroy, do not implicitly synchronize unrelated work, and allow provisioning workqueues to reduce false dependencies. Streams are created from a green context rather than relying on implicit device state, making targeting explicit. Benchmarks on a Blackwell GPU show a critical kernel completing in 0.007 ms with green context partitioning, compared to 0.140 ms with high-priority streams and 3.727 ms with equal priority. The tradeoff is fewer SMs for the bulk workload, but green contexts are opt-in and can be adopted incrementally alongside existing CUDA code.


### [Accelerating a ROS 2 Node with an AI Agent and NVIDIA Isaac ROS](https://yomu.fyi/post/accelerating-a-ros-2-node-with-an-ai-agent-and-nvidia-isaac-ros.md)
- Company: [NVIDIA](https://yomu.fyi/company/nvidia-developer-blog.md)
- Author: Tanya Lenz
- Published: Sep 22, 2026
- Type: Tutorial

GPU acceleration alone does not guarantee fast ROS 2 graphs because messages may still be serialized or copied through CPU memory. NVIDIA contributed the rosidl::Buffer abstraction and CUDA buffer backend to ROS 2 Lyrical, enabling zero-copy transport of GPU-resident payloads between co-located nodes when they share the same host, CUDA device, Linux user, and a supported RMW implementation, with automatic CPU fallback otherwise. An AI coding agent using the migrate-node-to-rosidl-buffer skill can audit an existing CUDA-accelerated node by tracing data movement, running a copy-boundary audit, and implementing a minimal interface-preserving patch. The tutorial demonstrates this on the Depth Anything 3 TensorRT node, updating subscription options and message storage to use CUDA buffer backend without changing the sensor\_msgs/msg/Image message definition. The resulting accelerated workload can be deployed on NVIDIA Jetson AGX Thor, preserving standard ROS 2 messages while avoiding unnecessary serialization and host copies.


### [Translating CUDA Tile Operations from Python to Rust Using Agentic AI](https://yomu.fyi/post/translating-cuda-tile-operations-from-python-to-rust-using-agentic-ai.md)
- Company: [NVIDIA](https://yomu.fyi/company/nvidia-developer-blog.md)
- Author: Tanya Lenz
- Published: Sep 16, 2026
- Type: Tutorial

cuTile Rust (cutile-rs) extends Rust’s ownership model to tile-based GPU kernels, and this post describes an agentic workflow for translating cuTile Python and Triton-TileIR implementations. Because all three front ends target CUDA Tile IR and share the tileiras compiler, the workflow compares reference and translated IR while making Python’s implicit specializations explicit through Rust const generics, fixed dtype symbols, and FFI checks. A bounded multi-agent pipeline covers analysis, device kernels, host and FFI code, correctness, and benchmarking, with machine-checkable verdicts controlling progress. It ported all 24 public TileGym operators, roughly 40 kernels, and reported 99.5% average cuTile Python performance across 347 paired configurations on NVIDIA DGX B200. The skill ships with TileGym and uses acceptance checks requiring correctness, backend execution, and a geomean speedup of at least 0.95.


### [How NVIDIA NVLink 6 Delivers Multi-Layer Resiliency for AI Factories](https://yomu.fyi/post/how-nvidia-nvlink-6-delivers-multi-layer-resiliency-for-ai-factories.md)
- Company: [NVIDIA](https://yomu.fyi/company/nvidia-developer-blog.md)
- Author: Elizabeth Goodman
- Published: Sep 15, 2026
- Type: Explainer

NVIDIA NVLink 6 is presented as a scale-up networking fabric for large AI factories, where dropped packets, link degradation, and node interruptions can reduce training goodput or inference capacity. It connects 72 Rubin GPUs in the Vera Rubin NVL72 into a single scale-up domain and uses a multi-layer resiliency stack spanning physical, link, application, software, and system layers. Physical Layer techniques include lightweight FEC, Physical Layer Retry (PLR), UPHY recovery, and hardware replay buffers, while credit-based flow control (CBFC) prevents injection when the next hop lacks buffer capacity. Redundant switch trays and management paths, local fault containment, NMX high availability, checkpointing, topology-aware recovery, and targeted switch maintenance are described as mechanisms for containing failures and preserving workloads; the post concludes that this integrated design targets continuous operation and maximum cluster productivity.


### [High-Throughput Structure Prediction with BioNeMo Inference Runtime](https://yomu.fyi/post/high-throughput-structure-prediction-with-bionemo-inference-runtime.md)
- Company: [NVIDIA](https://yomu.fyi/company/nvidia-developer-blog.md)
- Author: Elizabeth Goodman
- Published: Sep 10, 2026
- Type: Tutorial

Proteome-scale biomolecular structure prediction requires moving large worklists efficiently, and NVIDIA BioNeMo Inference Runtime (BioIR) accelerates supported models on NVIDIA GPUs while retaining a PyTorch workflow. Its end-to-end processor passes InputRequest objects through parsing, tokenization, feature generation, GPU inference, and PDB or mmCIF writing; a Ray backend places complete model replicas on GPUs for concurrent independent inputs, while direct PyTorch integration supports custom code. In a matched 1,000-target human-dimer benchmark, BioIR completed all targets at 58.5K successfully folded residues per allocated GPU-hour versus 20.2K for the tested public implementation, a 2.90× improvement, which ran out of memory on 29 targets. Rated-power-equivalent estimates for one million comparable targets were 11 versus 35 MWh at 8-GPU TDP and 21 versus 64 MWh at full-node maximum power, excluding data-center overhead and several pipeline costs.


### [To Infinity and Beyond: ThunderKittens Now on NVIDIA Vera Rubin NVL72!](https://yomu.fyi/post/to-infinity-and-beyond-thunderkittens-now-on-nvidia-vera-rubin-nvl72.md)
- Company: [Together AI](https://yomu.fyi/company/together-ai.md)
- Published: Sep 10, 2026
- Type: Explainer

Together’s kernels team ported ThunderKittens to NVIDIA Vera Rubin NVL72 and investigated why existing Blackwell GEMMs underuse the platform. The work targets NVFP4 and FP8 GEMMs: although Rubin preserves Blackwell’s programming model, the original kernels reach only about 42.1% and 44.4% of roofline because tensor cores consume operands twice as fast as the old pipeline supplies them. The team integrates Rubin’s 64-byte K step, exclusive tensor memory extending to 576 columns, oversized shared memory up to 328 KiB, B-side collector buffers, and earlier A-release signaling into its NVFP4 kernel. The resulting kernel exceeds 22 PFLOPS and is described as competitive with cuBLAS and CuTe DSL, while the post also compares Rubin’s specifications with NVIDIA HGX B200.


### [CUDA Toolkit 13.4 Adds Windows on Arm Support and Greater Control over Shared GPUs](https://yomu.fyi/post/cuda-toolkit-13-4-adds-windows-on-arm-support-and-greater-control-over.md)
- Company: [NVIDIA](https://yomu.fyi/company/nvidia-developer-blog.md)
- Author: Jonathan Bentz
- Published: Sep 9, 2026
- Type: Announcement

CUDA Toolkit 13.4 extends CUDA application support to Windows on Arm and provides preview support for NVIDIA Rubin (compute capability 107), alongside updates to GPU management, CUDA Python, CCCL, Nsight tools, and core math libraries. Multi-Process Service V3 adds a scriptable CLI, named server instances, namespaces, TOML configuration, streaming multiprocessor partition controls, and cgroup-integrated GPU memory limits for shared, containerized GPU workloads. CUDA Compute Fabric Transport lets communication-library developers address named NVLink endpoints and issue asynchronous put, get, and reduction operations without mapping every remote allocation into virtual address space; it is exposed only through the Driver API. The release also decouples driver installation from the toolkit, exposes locality and unified-memory residency queries, and changes coherent platforms to CDMM by default while retaining NUMA support.


### [Introducing CUDA Rust: Two Tracks for Writing GPU Kernels](https://yomu.fyi/post/introducing-cuda-rust-two-tracks-for-writing-gpu-kernels.md)
- Company: [NVIDIA](https://yomu.fyi/company/nvidia-developer-blog.md)
- Author: Elizabeth Goodman
- Published: Sep 8, 2026
- Type: Announcement

NVIDIA’s CUDA Rust effort targets the GPU-kernel gap in Rust-based systems, where host code can be Rust but kernels often require another language. It presents two tracks: cuda-oxide compiles #\[kernel\] functions through Rust MIR, Pliron, and LLVM IR to PTX for SIMT programming, while cutile-rs expresses computations as tiles and JIT-compiles embedded ASTs through CUDA Tile IR. The SIMT example uses DisjointSlice, typed thread indices, and launch contracts to constrain per-thread output access and validate launch configuration; Tile leaves thread mapping and shared-memory management to its compiler. The post recommends starting with Tile because the compiler decides tile mapping per architecture, and using SIMT when explicit thread or memory control is needed. Both projects are early-stage, with cuda-oxide in early alpha and cutile-rs further along but still incomplete and subject to API changes.


### [The Modern CUDA Toolbox in Practice: A Step-by-Step Optimization Walkthrough](https://yomu.fyi/post/the-modern-cuda-toolbox-in-practice-a-step-by-step-optimization-walkth.md)
- Company: [NVIDIA](https://yomu.fyi/company/nvidia-developer-blog.md)
- Author: Elizabeth Goodman
- Published: Sep 2, 2026
- Type: Tutorial

An image-processing pipeline serves as the test case for a modern CUDA toolbox, starting with RGB-to-grayscale conversion, 32-by-32 tile median computation, and host/device transfers. It advances through six incremental changes: adopting CCCL and Compute Sanitizer for indexing bugs, adding NVTX for Nsight Systems benchmarks, replacing hand-rolled work with CUB algorithms, using pooled device containers and pinned host containers, and assigning each image its own stream with asynchronous copies. The implementation uses cuda::stream, cuda::copy\_bytes, cuda::device\_buffer, and CUB calls on per-thread streams, allowing kernels and transfers to overlap instead of executing sequentially on the default stream. The reported total duration falls from 6.8 seconds to about 23 milliseconds for three images, a roughly 300x improvement, while the post characterizes the code as safer and easier to maintain without low-level optimizations.


### [ParallelKernelBench: Frontier LLMs can't write fast multi-GPU kernels (yet)](https://yomu.fyi/post/parallelkernelbench-frontier-llms-can-t-write-fast-multi-gpu-kernels-y.md)
- Company: [Together AI](https://yomu.fyi/company/together-ai.md)
- Published: Jun 23, 2026
- Type: Benchmark

ParallelKernelBench (PKB) measures whether language models can replace PyTorch + NCCL implementations with fast CUDA kernels for multi-GPU workloads, where communication can account for over 20% of inference latency. Its 87 problems come from real codebases and span tensor, context, data, expert, sequence, and FSDP/ZeRO parallelism, with hardware topology supplied and symmetric memory used for direct GPU communication over NVLink. In zero-shot evaluation, GPT-5.5 solved 28 problems and produced 22 faster-than-baseline solutions; sampling three attempts raised those figures to 36 and 27, but the best fast 1@3 result was only 31%. Models performed best on familiar collective primitives, tensor-parallel GEMMs, and Ulysses-style context parallelism, while failures often involved rank coordination, data partitioning, collective ordering, and underuse of TMA or NVLS.


### [Making FlashAttention-4 faster for inference](https://yomu.fyi/post/making-flashattention-4-faster-for-inference.md)
- Company: [Modal](https://yomu.fyi/company/modal.md)
- Author: Charles Frye
- Published: Jun 11, 2026
- Type: Explainer

FlashAttention-4, originally designed for pre-training, required modifications to efficiently handle LLM inference workloads dominated by the memory bandwidth-limited decode phase. Modal’s contributions include adjusting parallelism strategies from query parallelism to key/value parallelism, implementing split KV to parallelize across KV tiles, and adding a heuristic for optimal splits. They added support for FP8 inputs, arbitrary KV page sizes via cp.async loads, and reduced wasted work for small query sizes by using a single query tile. The kernel also gained extended GQA packing for irregular head ratios, using cp.async for query tiles to bypass TMA alignment restrictions. These changes, measured in throughput improvements up to 3x for single-token decode and up to 2.4x for small page sizes, demonstrate the leverage of flexible parallelism and tile-based programming for inference optimization.


### [How we achieved truly serverless GPUs](https://yomu.fyi/post/how-we-achieved-truly-serverless-gpus.md)
- Company: [Modal](https://yomu.fyi/company/modal.md)
- Author: Charles Frye
- Published: May 12, 2026
- Type: Problem & solution

GPU inference workloads suffer from spiky demand, leading to low GPU Allocation Utilization when capacity is fixed. Modal reduced replica startup latency from over 2 kiloseconds to about 50 seconds using four techniques. A buffer of idle GPUs removes instance allocation from the hot path. A custom content-addressed filesystem serves container images lazily from a multi-tier cache. CPU and CUDA checkpoint/restore fast-forward through initialization by directly restoring processes and GPU contexts into memory. These optimizations enable truly serverless GPU inference, as demonstrated by Reducto’s document processing scaling to thousands of GPUs with cold starts reduced from ~70s to ~12s.


### [Speeding up GPU kernels by 38% with a multi-agent system](https://yomu.fyi/post/speeding-up-gpu-kernels-by-38-with-a-multi-agent-system.md)
- Company: [Cursor](https://yomu.fyi/company/cursor.md)
- Author: Wilson Lin, Sahil Modi, Yuan Zhang & Edward Lin
- Published: Apr 14, 2026
- Type: Benchmark

The post reports a three-week experiment in which a multi-agent system optimized 235 CUDA kernel problems generated from workloads spanning more than 124 production open-source models. Using SOL-ExecBench on 27 NVIDIA Blackwell 200 GPUs, the system used a planner to distribute and rebalance work, while autonomous workers benchmarked, debugged, and refined kernels without developer intervention. It ran separate CUDA C with inline PTX and CuTe DSL runs, reaching assembly-level optimization and learning a documented higher-level API. Against a single-agent optimized PyTorch baseline, it outperformed 149 of 235 problems, achieving a 1.38x geometric mean ratio, while 45 solutions exceeded 2x. The highlighted cases included an 84% geomean gain over a human FlashInfer baseline for paged-prefill attention, a 39% gain for NVFP4 MoE linear, and a GEMM reaching 86% of a tuned cuBLAS baseline; median SOL was 0.56, leaving room for improvement.


### [1.5x faster MoE training with custom MXFP8 kernels](https://yomu.fyi/post/1-5x-faster-moe-training-with-custom-mxfp8-kernels.md)
- Company: [Cursor](https://yomu.fyi/company/cursor.md)
- Author: Stuart Sul
- Published: Aug 29, 2025
- Type: Benchmark

An upgrade from Hopper H100s to Blackwell B200s exposed the Mixture-of-Experts (MoE) layer, implemented with MegaBlocks, as a major training bottleneck: it consumed nearly 53% of forward-pass time and 27% of backward-pass time. The team rewrote the layer at the GPU-kernel level using CUDA, PTX, and some ThunderKittens, while moving from BF16 to microscaled FP8 (MXFP8) with nearly zero loss in training quality. The implementation addresses Blackwell-specific tensor memory constraints and quantization overhead, including custom MXFP8 quantization that sustains more than 6.2 TB/s and emits scale matrices directly compatible with tcgen05.mma. On an internal model, MoE execution became 3.5x faster in both passes, producing a 1.5x end-to-end speedup on Blackwell and 2x versus the original Hopper setup, measured in tokens per second per GPU.


### [What are these "GPUs" really?](https://yomu.fyi/post/what-are-these-gpus-really.md)
- Company: [Fly.io](https://yomu.fyi/company/fly-io.md)
- Published: Dec 11, 2023
- Type: Explainer

The post explains how graphics hardware evolved from CPU-offloaded video rendering into modern GPUs and why those devices now matter for AI/ML. It traces the path from the Commodore 64’s VIC-II and 3D accelerator cards through Quake, Tomb Raider, integrated graphics, and increasingly parallel hardware, while describing components such as framebuffers, command processors, shader hardware, memory, and display outputs. It then connects shader and CUDA cores to workloads including GPT-2, Stable Diffusion, and speech or text processing, and presents model chaining as more useful than treating models as isolated tools. A conference-video workflow combines ffmpeg, Whisper, subtitle segmentation, a large language model, Markdown, and human review to produce accessible written material. The conclusion distinguishes graphics-capable cards from A100 accelerators, which the post says cannot process graphics or encode video but are highly effective for AI/ML workloads.