Google Cloud C4 Brings a 70% TCO improvement on GPT OSS with Intel and Hugging Face

Summary

Intel and Hugging Face benchmarked OpenAI GPT OSS text generation to evaluate performance on Google Cloud C4 virtual machines powered by Intel Xeon 6 processors against previous-generation C3 instances. The benchmark evaluated the unsloth/gpt-oss-120b-BF16 Mixture of Experts model across batch sizes from 1 to 64 using bfloat16 precision, static KV cache, and SDPA attention. To eliminate redundant computation, Hugging Face merged an optimization directing each expert sub-network to process only its routed tokens rather than all tokens. Testing demonstrated that C4 instances deliver 1.4x to 1.7x higher throughput per vCPU compared to C3 instances. At a batch size of 64, C4 achieved a 1.7x Total Cost of Ownership improvement alongside lower hourly pricing.

Takeaways

1. 1

   Hugging Face merged an MoE optimization (PR #40304) that routes only designated tokens to each expert, eliminating redundant compute and reducing wasted FLOPs.

2. 2

   Google Cloud C4 instances with Intel Xeon 6 processors achieved 1.4x to 1.7x higher normalized throughput per vCPU over C3 instances across batch sizes up to 64.

3. 3

   At batch size 64, C4 delivered a 1.7x Total Cost of Ownership advantage over C3 due to combined throughput gains and near price parity per vCPU.