Loading…
Inside NVIDIA Rubin GPU Architecture: Powering the Era of Agentic AI
NVIDIA Developer BlogEduardo Alvarez
Summary
Agentic artificial intelligence workloads require sustained multistep inference, demanding low per-step latency, high decode throughput, and efficient long-context processing. NVIDIA designed the Rubin GPU to deliver up to ten times more agentic throughput per unit of energy than Blackwell. The hardware combines 336 billion transistors across two dies unified by NV-HBI, up to 288 GB of HBM4 memory providing 22 TB/s peak bandwidth, and the third-generation Transformer Engine reaching up to 50 petaflops of NVFP4 compute. Performance optimizations include inline descriptor updates for the Tensor Memory Accelerator, doubled K-dimension instruction throughput, 3-bit lookup-table weights, and structured 2:4 sparse attention compression. At the rack level, Vera Rubin NVL72 leverages Intelligent Power Smoothing and DSX MaxLPS to enable up to 40% more GPUs within a fixed power budget.
Context
Agentic AI workloads require continuous reasoning, tool use, and long-context processing across many steps, challenging data centers to provide high throughput and low latency within fixed power envelopes.
Approach / What changed
NVIDIA created the Rubin GPU and Vera Rubin NVL72 rack architecture, featuring dual-die packaging, HBM4 memory, expanded precision support, inline TMA updates, doubled K-dimension GEMM throughput, structured attention sparsity, and rack-scale power smoothing.
Takeaways
- The Rubin GPU integrates two dies over NV-HBI, providing 336 billion transistors, 224 streaming multiprocessors, 896 Tensor Cores, and up to 288 GB of HBM4 memory with 22 TB/s bandwidth.
- Matrix operations gain efficiency by doubling Tensor Core throughput along the K dimension, which reduces loop iterations and benefits context and decode GEMMs at high tensor-parallel scale.
- Vera Rubin NVL72 uses Intelligent Power Smoothing and DSX MaxLPS to absorb transient power swings, enabling operators to provision up to 40% more GPUs within the same power budget.
Related reading
NVIDIA Vera CPU Boosts AI Factory Throughput to Accelerate Agentic Workloads
Agentic AI workflows and reinforcement learning rely heavily on sequential CPU tasks, such as tool calls, sandboxed code execution, data processing, and environment rollouts between GPU steps. When CPU performance degrades under full socket load, training time inflates, serving latency grows, and GPU KV-caches face eviction. The NVIDIA Vera CPU addresses these bottlenecks by featuring 88 Olympus cores on a monolithic compute die, paired with power-efficient LPDDR5x memory and the Scalable Coherency Fabric. These cores incorporate a neural branch predictor, a 10-wide decode front end, deep out-of-order execution, and NVIDIA Spatial Multithreading. Consequently, Vera achieves 1.8x faster core execution, 40% lower peak loaded latency compared to x86 CPUs, and up to 1.2 TB/s total memory bandwidth, preventing costly KV-cache recomputations.
Michelle Horton