How to Size GPUs for AI Inference and TCO Without Overspending

Summary

GPU sizing for AI inference depends on use case, token patterns, latency targets, concurrency, cache hit rate, model choice and deployment strategy rather than raw hardware specifications alone. The post proposes a core-and-flex capacity model, combining baseline on-prem or reserved cloud capacity with elastic spot or on-demand GPUs for surges, launches and experiments. It also recommends matching GPU memory to prompt lengths and concurrency, while using quantization, pruning and distillation to reduce footprint and operating cost. A NeMo example prunes Qwen3-8B from 36 to 24 layers or narrows its dimensions toward a roughly 6B student before teacher-student distillation. In the illustrative run, width pruning reached lower final validation loss than depth pruning, 3.21 versus 3.60, while depth pruning converged faster.

Context

Organizations need to size GPU resources for inference while balancing latency targets, model choices, variable traffic patterns and budget constraints. Relevant measures include Time to First Token, 99th-percentile latency, intertoken latency, token lengths, concurrency, cache hit rate and requests per daily active user.

Approach / What changed

The post presents a workload-driven sizing framework covering use case, model, application scale, token patterns, latency, caching, traffic and contract length. It recommends core-and-flex capacity planning, GPU selection based on memory and concurrency, and model optimization through quantization, pruning and distillation. Its pruning example uses NVIDIA NeMo with Qwen3-8B to create roughly 6B students through depth or width changes, followed by distillation.

Takeaways

• A core-and-flex strategy uses on-prem or reserved cloud GPUs for steady demand and spot or on-demand cloud capacity for surges, launches and experiments.
• The NeMo pruning example reduces Qwen3-8B through either 36-to-24-layer depth pruning or width reductions from 12288 to 9216 FFN size and 4096 to 3584 hidden size.
• In the illustrative validation run, width pruning ended at loss 3.21 versus 3.60 for depth pruning, while depth pruning converged faster.

Related reading