Scaling-up BERT Inference on CPU (Part 1)

Summary

Scaling BERT-like model inference on CPUs presents practical deployment challenges as transformer architectures proliferate in production environments. Using a reproducible Hydra-based benchmarking framework, inference workloads were evaluated on an Amazon Web Services c5.metal instance featuring an Intel Xeon Platinum 8275 processor with AVX512 and VNNI support. Out-of-the-box evaluations showed PyTorch 1.8.1 delivering lower latency than Google TensorFlow 2.4.0, likely due to PyTorch relying on OpenMP and Intel MKL/oneDNN rather than Eigen. Implementing multiple inference streams by pinning parallel model instances to dedicated CPU cores yielded near-linear throughput scaling. Matching instance core allocations to specific problem sizes also demonstrated that smaller workloads can achieve optimal latency on fewer cores, enabling up to a sixfold reduction in cloud compute costs.

Context

As transformer architectures grow in production adoption, efficiently deploying and executing BERT-like models at scale on modern CPU architectures remains a technical challenge.

Approach / What changed

A benchmarking suite built with Facebook AI Research's Hydra was run on an AWS c5.metal bare-metal instance (Intel Xeon Platinum 8275 CPU, 48 cores/96 threads) running Ubuntu 20.04 LTS. The evaluation compared baseline PyTorch 1.8.1 and TensorFlow 2.4.0 configurations, and assessed parallel execution using multiple inference streams where independent model instances are pinned to non-overlapping subsets of CPU cores.

Takeaways

• Out of the box, PyTorch 1.8.1 demonstrated lower inference latency than TensorFlow 2.4.0 across tested configurations, which may stem from PyTorch using OpenMP and Intel MKL/oneDNN for linear algebra while TensorFlow uses Eigen.
• Running multiple parallel inference streams by allocating dedicated, non-overlapping CPU cores to independent model instances achieved near-linear throughput scalability.
• Smaller workloads such as short sequences and small batch sizes require fewer CPU cores for optimal latency, allowing an 8-core machine at $0.808/hour to achieve a 6x cost reduction over a 48-core machine at $4.848/hour.

Related reading

• huggingface.co · Jan 13, 2022

  Case Study: Millisecond Latency using Hugging Face Infinity and modern CPUs

  Deploying large Transformer models at scale often faces severe prediction latency bottlenecks, driving up infrastructure costs and limiting real-time production use cases. To address this challenge, Hugging Face evaluated Infinity, a containerized hardware-optimized inference solution paired with the Infinity Multiverse model optimization service. Testing covered 192 configurations on Amazon EC2 C6i instances powered by 3rd generation Intel Xeon Scalable processors across varying CPU cores, sequence lengths, and batch sizes. The benchmarks demonstrated that an Ice Lake-optimized DistilBERT container achieved up to 800% higher throughput than vanilla Transformers and delivered 1 to 4 millisecond end-to-end latencies for sequence lengths up to 64 tokens. Although Infinity was later discontinued in favor of Inference Endpoints and Optimum libraries, the results demonstrated substantial efficiency gains on modern CPU hardware.

  Philipp Schmid, Jeff Boudier, Morgan Funtowicz

• Hugging Face · Sep 29, 2025

  Accelerating Qwen3-8B Agent on Intel® Core™ Ultra with Depth-Pruned Draft Models