Loading…
NVIDIA
Technology company that designs powerful computer chips used for video games, data centers, and artificial intelligence.
- 155 posts summarised
- latest
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
From their blog
Latest articles
Giga-Scale AI and the Ethernet Evolution: How Spectrum-X Ethernet Rewrites the Rules
Generative AI training across hundreds of thousands of GPUs makes the scale-out network a first-order performance bottleneck, because synchronized collectives expose weaknesses in conventional Ethernet. Static ECMP, packet loss controls such as PFC, and slowly tuned congestion control can create stragglers, head-of-line blocking, latency spikes, and multi-tenant interference. Spectrum-X Ethernet addresses these conditions with hardware-accelerated adaptive per-packet routing, targeted ECN- and RTT-based congestion control, and a SuperNIC Plane Load Balancer that distributes traffic across independent network planes. Spectrum-X Multiplane uses shallow two-tier fat trees and host-edge optical devices to provide path diversity while scaling a two-tier design to over 128,000 endpoints. In the cited tests, Spectrum-X sustained 98% of theoretical line rate across GPU pairs, held P99 tail latency to 8–9 microseconds under 75% load, recovered from a link flap in 2.68 milliseconds, and limited bandwidth loss under 10% fabric failure to 11%.
NVIDIA Vera Rubin and Blackwell Set a New Standard for Agentic AI Performance per Watt
Agentic AI inference has shifted from single-turn requests to long, stateful workflows that reason, invoke tools, coordinate subagents, and reuse growing context, increasing token and power demands. The post presents SemiAnalysis AgentX, an InferenceX benchmark that replays prerecorded Claude Code sessions turn by turn, preserving context, input/output lengths, reasoning time, tool-call latency, and KV-cache pressure while varying concurrency. It evaluates tokens per megawatt alongside E2E normalized interactivity, standard interactivity, E2E latency, and TTFT; NVIDIA-measured Vera Rubin NVL72 results, pending SemiAnalysis review, show up to 30x higher throughput per megawatt than GB300 NVL72 at 160 tokens per second per user. GB300 also delivers up to 15x H200 throughput per megawatt, up to 10x lower cost per million tokens, and roughly 80x higher throughput on Kimi K3 2.8T, supported by MoE runtimes, communication overlap, Dynamo, and NVLink.
Maximizing AI Factory Performance per Watt with NVIDIA DSX MaxLPS
Power-constrained AI factories measure efficiency by AI output per watt, because facility overhead, rack losses, and operational inefficiency reduce the power available to compute. In an illustrative 100 MW budget, 60% reaches AI load, while static rack provisioning can strand headroom that another rack could use. NVIDIA DSX MaxLPS addresses this with dynamic power allocation, workload-profile power optimization, and 45° C thermal and site design, including warm-water liquid cooling. Its Dynamic Power Software, currently in Developer Preview, models topology from utility to GPUs and reallocates unused capacity within policy-defined budgets, with optional DSX Exchange integration. For Vera Rubin NVL72, NVIDIA projects up to 40% more Rubin GPU capacity within the same power budget, while validation compares unmanaged and managed runs using throughput, latency, errors, power, utilization, and compliance.
NVIDIA BlueField-4 Powers New Scale-In Network Infrastructure for Agentic AI Factories
Agentic AI factories connect diverse users, agents, applications, data sources, and storage systems to massively accelerated compute, making dedicated DPU processing essential for line-rate networking, storage, and security. Scale-In evolves north-south access into a coordinated infrastructure domain built with BlueField-4, NVIDIA DOCA, and Spectrum-X Ethernet. BlueField-4 combines programmable control-plane processing with inline data-path engines, a 64-core Grace CPU, LPDDR5X memory, PCIe Gen6 host connectivity, and an 800 Gb/s network interface. The architecture offloads security, tenant isolation, storage protocols, provisioning, and telemetry from host CPUs; the source states that Spectrum-X storage paths can provide up to 1.45x the throughput of off-the-shelf Ethernet, while independent control and observability support more consistent operations as AI infrastructure scales.
Solving Agentic AI Fleet Challenges with NVIDIA Vera CPU
Agentic AI fleets must handle unpredictable trajectories in which long sequential reasoning chains are interrupted by short bursts of parallel tool calls, retrieval, sandbox, or sub-agent work. Telemetry from 163,594 sessions found over 97% had unique trajectory profiles, making fleet sizing around several specialized CPU designs impractical; the sequential path is latency-bound while fan-out requires concurrency. It presents NVIDIA Vera CPU as a single balanced design point, combining strong loaded per-core performance, concurrency, memory bandwidth, and a low-latency monolithic architecture, with Olympus cores using a wide front end, branch prediction, and deep out-of-order execution. Using estimated SPEC CPU® 2026 results across four agentic workloads, it reports up to 1.5x the agentic performance of AMD Venice, while noting that Venice figures were estimated from SPECrate® 2026 data and that results may vary.
How NVIDIA Groq 3 LPX Unlocks Ultrafast Interactivity at Long Context on NVIDIA Vera Rubin
Serving agentic AI at more than 3,000 output tokens per second while retaining a 100K-token KV cache creates a small-batch tensor-parallelism challenge: fixed communication and coordination costs can offset distributed-compute gains. NVIDIA Groq 3 LPX addresses this with compiler-scheduled execution across 256 LP30 local processing units, 128 GB of collective SRAM, and 96 112-Gbps C2C links per chip, planning transfers before execution and overlapping communication with computation at 320-byte-vector granularity. Artificial Analysis measured 3,431 output tokens per second at 100K context on Gemma 4 31B, and 3,382 tokens per second at 10K context. On SPEED-Bench coding tasks, the same system reached a median 4,767 tokens per second and a P80 of 5,520. The post describes pairing LPX with Vera Rubin NVL72 through prefill-decode, attention-FFN, or external-drafter speculative-decoding configurations for projected large-model agentic workloads.
GPU-Accelerated Clustering for Financial Instruments at Scale
Quantitative strategies need instrument groupings that capture both ordinary co-movement and dependence during market stress, while remaining stable and inexpensive to recompute. The workflow constructs rolling absolute Pearson-correlation and tail pairwise dependence matrices (TPDM), factorizes them with nonnegative SymNMF using AdaptGrow, and derives soft loadings, hard argmax labels, and structural-break diagnostics. A trace-based formulation reduces estimated peak storage from approximately 20n2 to 4n2 bytes, while row-sharded distributed execution communicates O(nk) data; the system uses eigenspectrum-based selection between full-batch and block-stochastic AdaGrad. Across 250 synthetic windows, the setup tests planted membership changes and a joint tail-stress episode: ARI is intended to identify reclassification, whereas TPDM reveals co-crash dependence that ordinary correlation can miss. Reported factorization timings include 13.0 and 12.4 seconds for 100,000 instruments across four GB200 GPUs, and approximately two and four minutes for one million instruments across 64 GPUs on 16 nodes, not end-to-end timings.
NVIDIA AVO Reaches 100% on ARC-AGI-3, Demonstrating a Frontier-Level General-Purpose Architecture for Long-Horizon Autonomous Agents
NVIDIA’s Agentic Variation Operators (AVO) is a general-purpose agent architecture designed to make frontier language models reliable on extended, multistep tasks. It combines tool use and execution with persistent memory, an iterative inspect-plan-implement-evaluate loop, and a supervisor that can redirect work when progress stalls. In a seven-day GPU-kernel optimization study, AVO explored more than 500 directions, produced 40 committed versions, and outperformed cuDNN by up to 3.5% and FlashAttention-4 by up to 10.5% across evaluated configurations. On the ARC-AGI-3 public set, the same architecture achieved a 100.00 RHAE score across 25 environments, completing all 183 levels in 6,624 actions; the post cautions that comparisons with other systems are not controlled ablations.
Where Security Fits in an AI Agent Stack
The post explains where security controls should sit in an emerging AI agent stack as agents gain capability and operate over longer horizons. It distinguishes behavioral controls in models and harnesses, which guide proposed actions, from authoritative infrastructure controls that bind identity, enforce policy, contain failures, and record activity. The stack is presented as layers for models, harnesses, meta-harnesses, secure runtimes such as NVIDIA OpenShell, and inference infrastructure, with the runtime boundary established before launch so tools, plugins, MCP processes, and delegated runtimes operate under policy. It recommends least privilege, isolation, explicit authorization, auditability, short-lived and task-scoped access, independent checks, and safer failure states, while describing four security profiles from isolated to adversarial workloads. The conclusion is that every high-impact external effect must pass through enforcement layers the agent cannot bypass.
How Generative Recommenders Are Redefining RecSys at Scale
Generative recommenders (GRs) recast recommendation as next-item or next-action sequence prediction conditioned on user histories, replacing traditional embedding-similarity objectives and potentially unifying retrieval and ranking. The post presents HSTU, which uses timestamp-ordered interleaved items and actions, SiLU-based attention weighting, relative attention bias, and gating, alongside Semantic IDs, which derive a smaller vocabulary through hierarchical clustering of item embeddings. For production, recsys-examples combines TorchRec, DynamicEmb, Megatron-Core, recommender-specific KV and decoding components, while nv-embedding-cache provides tiered GPU, CPU, and remote embedding storage. On an H100, the Semantic ID path was reported as 1.85x higher in online throughput and 46% lower median latency than SGLang under the stated long-context, beam-search workload, and DLRM v3 reached 99,997 queries/sec.
Developing NVIDIA Holoscan Applications with CLI, Skills, and AI Coding Agents
NVIDIA Holoscan is a platform for real-time AI applications at the edge, and HoloHub provides reference applications and components. This post describes an engineer-guided workflow in which a coding agent builds a separate Python endoscopic tool segmentation application from the existing MONAI model, sample video, and Holoscan patterns. The Holoscan CLI, invoked through ./holohub, exposes shared commands for scaffolding, running, modes, tests, and benchmarking, while development skills and documentation guide the agent through project-specific operations. The application connects video replay, preprocessing, TensorRT inference, segmentation postprocessing, telemetry, and HoloViz visualization, with visual, smoke, and benchmark modes; the post reports that the CLI-plus-skills-plus-docs/examples configuration produced a separate application in 40 minutes using 11M tokens, and concludes that this combination offered the lowest resource overhead among the compared configurations.
Building Federated Multimodal AI Workflows with NVIDIA FLARE
Federating vision-language models (VLMs) across institutions requires keeping images, text, and prompts local while coordinating training and combining updates from sites with different task and modality mixes. NVIDIA FLARE addresses this workflow through server-scheduled rounds, client-local execution, and an explicit update contract, while supporting large-object externalization, pull-based tensor streaming, and disk-backed aggregation for large payloads. The post presents FedUMM as an adapter-based design in which clients train LoRA adapters over a frozen BLIP3o backbone and send only adapter updates, while full-model training remains possible when required. In reported eight-client experiments, adapter-only federation reduced per-client communication from 28.6 GB to 0.094 GB per round, improved VQA v2 by 0.7 points, and retained about 97% of centralized performance on VQA v2 and GenEval; the evaluation used simulated sites and does not establish clinical performance or formal privacy guarantees.
Post-Train NVIDIA Cosmos 3 Edge for On-Device Robot Control
Deploying large world models on robots is constrained by onboard memory and the control latency needed for continuous manipulation. The tutorial post-trains NVIDIA Cosmos 3 Edge, a 4B omni-model with a 2B NVIDIA Nemotron-based reasoner, into a DROID action policy using 76,000 successful teleoperated trajectories, then serves it through an OpenPI-compatible WebSocket server on Jetson Thor. The configuration predicts 32 future 8-dimensional joint-position actions at 15 Hz, while a receding-horizon loop executes a prefix and replans from fresh camera and robot-state observations. On an NVIDIA Jetson AGX Thor T5000, inference takes about 1.53 seconds per action chunk covering roughly 2.13 seconds of motion, and the policy reaches 22.9% success in closed-loop RoboLab evaluation; the source contrasts this with Cosmos 3 Nano’s 36.8%.
Evaluating AI Agent Skill Performance with NVIDIA SkillEvaluator
NVIDIA SkillEvaluator measures whether packaged agent skills improve agent performance by comparing task runs with and without a skill installed. It combines three evaluation tiers: static safety and structure checks, embedding-based distinctiveness analysis, and live evaluations in isolated Harbor sandboxes. In the August 12, 2026 benchmark snapshot, more than 300 verified skills spanning over 30 NVIDIA products were tested across Claude Code and Codex, with macro-averaged Skill Lift of +31 points across all dimensions and +39 excluding Security. Correctness rose from 46 to 87 and Effectiveness from 39 to 78, while Security moved from 97 to 98; token and runtime effects varied by skill, with one reducing both and another increasing them.
How AI Coding Agents Can Unlock Materials Simulation with NVIDIA ALCHEMI Toolkit
Atomistic simulation combines scientific judgment, compute-efficient implementation, and an accessible interface to the simulation stack, but the MLIP ecosystem lacks the mature interfaces available for classical force fields. NVIDIA ALCHEMI Toolkit addresses implementation with composable, PyTorch-native, GPU-accelerated building blocks and in-flight batching, while its agent skills and reference files supply API patterns to coding agents. The post benchmarks 45 generated pipelines across silicon equation of state, copper adsorption, and lithium self-diffusion, checking code features and running representative workloads on NVIDIA H200 GPUs. All 45 scripts used batched GPU execution, 38 completed screening, and all 15 production representatives completed; however, agents did not assess physical validity, and thermostat choices could suppress measured lithium diffusion by 3–5×, so independent validation against DFT, experiment, or another reference remains necessary.
Run Massive-Scale UMAP in Minutes Using Multiple GPUs—Without Losing Accuracy
UMAP workflows on tens to hundreds of millions of vectors are constrained by the cost of constructing an all-neighbors k-nearest-neighbor graph, especially when repeated runs support exploration and parameter tuning. The NVIDIA cuML and cuVS 25.06 feature distributes this graph-construction stage across multiple GPUs while retaining the out-of-core design that partitions data into balanced clusters, overlaps vectors across nearby clusters, computes local graphs, and merges them into a global graph. knn_n_clusters and knn_overlap_factor expose the memory, runtime, and graph-quality trade-off: more clusters reduce per-GPU data, while more overlap can improve nearest-neighbor recall at higher computational and memory cost. On eight NVIDIA H100 GPUs, the reported evaluation found up to 74x end-to-end speedup over a projected CPU runtime for the 106M-vector MIRACL dataset, completing it in 8 minutes. The post states that performance continues to improve from one to eight GPUs while preserving comparable embedding quality across configurations.
Developing Nemotron 3.5 Lightning NVFP4 with QAD Using NVIDIA Model Optimizer
NVIDIA’s post presents a workflow for developing an NVFP4 checkpoint of the Nemotron 3.5 Lightning 30B-A3B model, targeting lower memory use and higher inference throughput without sacrificing accuracy. The method starts with post-training quantization (PTQ), producing a W4A16 student from the BF16 teacher, then applies quantization-aware distillation (QAD), where simulated quantization and KL-divergence loss align student logits with the frozen teacher. For the selected four_over_six recipe, 32K calibration performed best, and the recipe using W4A16 Mamba linears offered the best reported trade-off between accuracy degradation and inference performance; the checkpoint is 22 GB versus 66 GB and delivers up to 4x faster throughput. The workflow includes Model Optimizer examples and launchers for Megatron-LM and Megatron-Bridge, covering calibration, distillation, evaluation, and export.
Serve Qwen3.8-2.4T-A95B, a 2.4T-Parameter Model, with Configurable Reasoning on NVIDIA GB300 NVL72
Alibaba’s Qwen3.8-2.4T-A95B (Qwen3.8-Max) is an open-weight 2.4T-parameter model with 95B activated per token, designed for reasoning and agentic workloads. Its fine-grained mixture-of-experts architecture combines full and linear attention, supporting context of up to one million tokens and output lengths up to 128K while using bounded recurrent state in linear-attention layers. NVIDIA describes multinode serving on the 72-GPU GB300 NVL72, where a 130 TB/s NVLink domain supports expert communication; in FP8, the model reportedly exceeds 4K tokens per second per GPU and 350 tokens per second per user without additional tuning. Developers can control reasoning depth with low, high, or xhigh modes, choose SGLang, vLLM, NVIDIA Dynamo, or NIM for serving, and use NeMo AutoModel for full SFT or LoRA post-training.
How to Choose Full-Stack Observability for NVIDIA AI Factories
AI infrastructure spans compute, networking, storage, orchestration, and applications, so a symptom at one layer may originate elsewhere. The post proposes selecting observability tools by failure domain, mapping components to telemetry sources, and reducing signals to a top-k alert set in a unified Prometheus/Grafana triage dashboard. For a DGX cluster using InfiniBand, BCM, and Slurm, it recommends Redfish/IPMI, DCGM, NVSM, UFM, and BCM, while deferring NetQ, NMX, Run:ai, and NIM until their corresponding capabilities are needed. The framework emphasizes DCGM for GPU metrics, NVSM for DGX system health, UFM for InfiniBand, and BCM for cluster and job aggregation, with specialized tools retaining low-level telemetry. An elevated InfiniBand bit error rate illustrates how retransmissions can slow one rank during NCCL all-reduce, causing synchronized training throughput to follow the slowest rank; alerts should connect concise signals to owners and remediation actions.
NVIDIA JetPack 7.2.1 Adds Agentic Video Skills and T3000 Emulation
JetPack 7.2.1 adds support for PyNvVideoCodec 2.2, NVIDIA's Python library for hardware-accelerated video encoding and decoding on GPUs, alongside foundational agentic video skills for Jetson workflows. These skills inspect the live target, query codec capabilities, generate Video Codec SDK or PyNvVideoCodec recipes, execute runs, and return measurements, warnings, and reproducible evidence. PyNvVideoCodec keeps frames in GPU-resident device memory through DLPack and CUDA device buffers, with multi-mode frame sampling and a ThreadedDecoder that pre-decodes frames in a background thread. The release also enables emulation of recently announced Jetson T3000 performance on a Jetson T5000 module in the Jetson Thor AGX Developer Kit, while the video skills are limited to Video Codec SDK and PyNvVideoCodec stages.
Topics they write about
All topics →Weekly digest
The best engineering reads, once a week.