# Maximize AI Factory Energy Efficiency Through Full-Stack Inference and Training Optimizations

[NVIDIA Developer Blog](https://yomu.fyi/company/nvidia-developer-blog) · Sachin Idgunji · Jun 23, 2026

**Type:** Explainer

## Summary

Power costs comprise up to 40% of AI factory operating expenses, making performance per watt critical within fixed site power limits. Operators maximize throughput by combining architectural enhancements, narrow-precision formats like NVFP4, and mixture-of-experts model designs that activate fewer parameters per token. During distributed LLM training in Megatron-LM, tuning individual GPU speeds along critical execution paths eliminates idle power waste without extending total training time. Additionally, the NVIDIA DSX platform orchestrates dynamic power allocation, 45°C direct-to-chip liquid cooling, and grid integration across racks and facilities. Together, these full-stack optimizations enable performance-optimized AI factories to deliver up to 2.6 times more tokens per second per megawatt.

## Context

Power can account for 40% of an AI factory's operating expenses, and most sites operate under fixed power limits imposed by regional providers. Under these constraints, performance per watt directly determines token generation costs, overall throughput, and revenue potential.

## Approach / What changed

Implementing full-stack optimizations across hardware, software, and facilities: utilizing narrow numerical formats like NVFP4, adopting mixture-of-experts architectures, tuning GPU speeds along the training critical path in Megatron-LM, deploying direct-to-chip liquid cooling, and orchestrating dynamic power allocation and grid integration with NVIDIA DSX.

## Takeaways

- Narrow-precision formats such as NVFP4 deliver higher tokens per second per watt than FP8 at equivalent accuracy across fixed power envelopes.
- Tuning individual GPU speeds during distributed training with Megatron-LM reduces idle synchronization waste, achieving up to roughly 25% energy savings at similar iteration step times.
- NVIDIA DSX pairs 45°C liquid cooling, dynamic power allocation, and grid-aware orchestration to recover stranded power and achieve up to 2.6x higher tokens per second per megawatt.

**Tags:** [Architecture](https://yomu.fyi/topic/architecture), [LLMs](https://yomu.fyi/topic/llm), [Machine Learning](https://yomu.fyi/topic/machine-learning), [Performance](https://yomu.fyi/topic/performance), [Scalability](https://yomu.fyi/topic/scalability)

[Read original post](https://developer.nvidia.com/blog/maximize-ai-factory-energy-efficiency-through-full-stack-inference-and-training-optimizations)