Case Study: Millisecond Latency using Hugging Face Infinity and modern CPUs

Summary

Deploying large Transformer models at scale often faces severe prediction latency bottlenecks, driving up infrastructure costs and limiting real-time production use cases. To address this challenge, Hugging Face evaluated Infinity, a containerized hardware-optimized inference solution paired with the Infinity Multiverse model optimization service. Testing covered 192 configurations on Amazon EC2 C6i instances powered by 3rd generation Intel Xeon Scalable processors across varying CPU cores, sequence lengths, and batch sizes. The benchmarks demonstrated that an Ice Lake-optimized DistilBERT container achieved up to 800% higher throughput than vanilla Transformers and delivered 1 to 4 millisecond end-to-end latencies for sequence lengths up to 64 tokens. Although Infinity was later discontinued in favor of Inference Endpoints and Optimum libraries, the results demonstrated substantial efficiency gains on modern CPU hardware.

Context

Deploying complex, large Transformer models in production at scale is constrained by prediction latency bottlenecks, which makes large deployments expensive and real-time use cases impractical.

Approach / What changed

Hugging Face deployed Infinity, a containerized inference architecture consisting of the Infinity Container and Infinity Multiverse optimization service. They benchmarked an end-to-end pipeline (pre-processing, prediction, post-processing) using a DistilBERT sequence classification model on Amazon EC2 C6i instances with 3rd generation Intel Xeon Scalable processors (Ice Lake). Testing evaluated 192 configurations across physical CPU cores (1, 2, 4, 8), sequence lengths (8 to 512), and batch sizes (1 to 32).

Takeaways

• The Ice Lake-optimized Infinity Container achieved up to 800% higher throughput and lower latency than vanilla Transformers on Ice Lake, and up to 34% improvement compared to Cascade Lake instances.
• When running on 2 physical cores with batch size 1, Infinity achieved throughput ranging from 248 requests per second at sequence length 8 to 12 requests per second at sequence length 512.
• Infinity achieved 1 to 4 millisecond latencies for sequence lengths up to 64 tokens and maintained minimal deviation between average latency and p95, p99, or maximum latency percentiles.

Related reading

• huggingface.co · Apr 20, 2021

  Scaling-up BERT Inference on CPU (Part 1)

  Scaling BERT-like model inference on CPUs presents practical deployment challenges as transformer architectures proliferate in production environments. Using a reproducible Hydra-based benchmarking framework, inference workloads were evaluated on an Amazon Web Services c5.metal instance featuring an Intel Xeon Platinum 8275 processor with AVX512 and VNNI support. Out-of-the-box evaluations showed PyTorch 1.8.1 delivering lower latency than Google TensorFlow 2.4.0, likely due to PyTorch relying on OpenMP and Intel MKL/oneDNN rather than Eigen. Implementing multiple inference streams by pinning parallel model instances to dedicated CPU cores yielded near-linear throughput scaling. Matching instance core allocations to specific problem sizes also demonstrated that smaller workloads can achieve optimal latency on fewer cores, enabling up to a sixfold reduction in cloud compute costs.

  Morgan Funtowicz

• huggingface.co · Nov 4, 2021

  Scaling up BERT-like model Inference on modern CPU - Part 2

  Scaling transformer inference on modern CPUs requires coordinating hardware capabilities with software optimizations across memory allocation, parallel execution, and mathematical kernel libraries. Intel Ice Lake Xeon processors deliver up to seventy-five percent faster inference across natural language processing tasks compared to Cascade Lake by combining Sunny Cove architecture improvements with dedicated extensions like Intel Extension for PyTorch. Fine-tuning software knobs such as alternative allocators, threading runtimes like Intel OpenMP, and core allocations can involve thousands of possible configuration combinations. Rather than evaluating these combinations through exhaustive brute force, Bayesian optimization via Intel SigOpt achieves comparable latency within roughly twenty trials and reveals experiment-dependent parameter importance. These software tuning strategies enable low-latency production transformer serving without dedicated accelerators.