Scaling up BERT-like model Inference on modern CPU - Part 2

Summary

Scaling transformer inference on modern CPUs requires coordinating hardware capabilities with software optimizations across memory allocation, parallel execution, and mathematical kernel libraries. Intel Ice Lake Xeon processors deliver up to seventy-five percent faster inference across natural language processing tasks compared to Cascade Lake by combining Sunny Cove architecture improvements with dedicated extensions like Intel Extension for PyTorch. Fine-tuning software knobs such as alternative allocators, threading runtimes like Intel OpenMP, and core allocations can involve thousands of possible configuration combinations. Rather than evaluating these combinations through exhaustive brute force, Bayesian optimization via Intel SigOpt achieves comparable latency within roughly twenty trials and reveals experiment-dependent parameter importance. These software tuning strategies enable low-latency production transformer serving without dedicated accelerators.

Context

Maximizing deep learning inference efficiency on CPUs requires navigating complex software and hardware interactions, as standard setups often leave underlying multi-core architectures and low-level instruction sets underutilized.

Approach / What changed

Evaluating Intel oneAPI software components including Intel OpenMP, oneDNN, and framework extensions alongside memory allocators such as jemalloc, mimalloc, or tcmalloc, while utilizing Intel SigOpt Bayesian optimization to tune configuration parameters across sequence lengths and core counts.

Takeaways

• Intel Ice Lake Xeon processors achieve up to 75% faster inference on natural language processing tasks compared to Cascade Lake processors due to Sunny Cove architecture features and software enhancements.
• Intel oneAPI provides domain-specific libraries such as Intel oneDNN for neural network primitives, Intel oneMKL for linear algebra, and Intel OpenMP for high-performance multi-core parallelization.
• Using Intel SigOpt for Bayesian optimization allows tuning parameter spaces with up to 1,920 configurations in roughly 20 to 25 trials while keeping latency within 8.6% of brute-force search results.

Related reading

• huggingface.co · Jun 15, 2022

  Intel and Hugging Face Partner to Democratize Machine Learning Hardware Acceleration

  Intel has joined Hugging Face's Hardware Partner Program to accelerate Transformer training, fine-tuning, and inference on Intel platforms. Large Transformer models introduce latency bottlenecks in production workloads like search and chatbots, where hardware-level optimization typically requires tedious trial and error. To streamline model optimization, the collaboration introduces Optimum Intel, an open-source library integrating the Intel Neural Compressor for automated quantization, pruning, and distillation. A demonstration applies post-training dynamic quantization to a fine-tuned DistilBERT classification model using a CPU-only PyTorch setup. The quantized model converted 38 Linear and 2 Embedding operators to 8-bit integers, reducing evaluation duration by 1.34x while keeping the accuracy drop within a 5% threshold.

  Julien Simon

• huggingface.co · Jan 13, 2022

  Case Study: Millisecond Latency using Hugging Face Infinity and modern CPUs

  Deploying large Transformer models at scale often faces severe prediction latency bottlenecks, driving up infrastructure costs and limiting real-time production use cases. To address this challenge, Hugging Face evaluated Infinity, a containerized hardware-optimized inference solution paired with the Infinity Multiverse model optimization service. Testing covered 192 configurations on Amazon EC2 C6i instances powered by 3rd generation Intel Xeon Scalable processors across varying CPU cores, sequence lengths, and batch sizes. The benchmarks demonstrated that an Ice Lake-optimized DistilBERT container achieved up to 800% higher throughput than vanilla Transformers and delivered 1 to 4 millisecond end-to-end latencies for sequence lengths up to 64 tokens. Although Infinity was later discontinued in favor of Inference Endpoints and Optimum libraries, the results demonstrated substantial efficiency gains on modern CPU hardware.