Loading…
Maximize AI Factory Energy Efficiency Through Full-Stack Inference and Training Optimizations
NVIDIA Developer BlogSachin Idgunji
Summary
Power costs comprise up to 40% of AI factory operating expenses, making performance per watt critical within fixed site power limits. Operators maximize throughput by combining architectural enhancements, narrow-precision formats like NVFP4, and mixture-of-experts model designs that activate fewer parameters per token. During distributed LLM training in Megatron-LM, tuning individual GPU speeds along critical execution paths eliminates idle power waste without extending total training time. Additionally, the NVIDIA DSX platform orchestrates dynamic power allocation, 45°C direct-to-chip liquid cooling, and grid integration across racks and facilities. Together, these full-stack optimizations enable performance-optimized AI factories to deliver up to 2.6 times more tokens per second per megawatt.
Context
Power can account for 40% of an AI factory's operating expenses, and most sites operate under fixed power limits imposed by regional providers. Under these constraints, performance per watt directly determines token generation costs, overall throughput, and revenue potential.
Approach / What changed
Implementing full-stack optimizations across hardware, software, and facilities: utilizing narrow numerical formats like NVFP4, adopting mixture-of-experts architectures, tuning GPU speeds along the training critical path in Megatron-LM, deploying direct-to-chip liquid cooling, and orchestrating dynamic power allocation and grid integration with NVIDIA DSX.
Takeaways
- Narrow-precision formats such as NVFP4 deliver higher tokens per second per watt than FP8 at equivalent accuracy across fixed power envelopes.
- Tuning individual GPU speeds during distributed training with Megatron-LM reduces idle synchronization waste, achieving up to roughly 25% energy savings at similar iteration step times.
- NVIDIA DSX pairs 45°C liquid cooling, dynamic power allocation, and grid-aware orchestration to recover stranded power and achieve up to 2.6x higher tokens per second per megawatt.
Related reading
Boost Inference Performance up to 15x on NVIDIA Blackwell Using DFlash Speculative Decoding
Sequential token generation in autoregressive LLMs limits GPU utilization and serving throughput, while conventional speculative decoding remains constrained by autoregressive draft models. DFlash addresses this bottleneck by employing a lightweight block-diffusion drafter that predicts an entire block of masked candidate tokens in a single forward pass. The architecture incorporates target hidden-state conditioning and key-value injection into draft model projections while relying on the target model for parallel verification. Evaluated on NVIDIA Blackwell GPUs across TensorRT-LLM, vLLM, and SGLang, DFlash increases gpt-oss-120b throughput by more than 15x at high interactivity targets and delivers up to 5.8x speedups on Gemma 4 31B compared to autoregressive decoding.
Amr Elmeleegy