# Scaling-up BERT Inference on CPU (Part 1)

huggingface.co · Morgan Funtowicz · Apr 20, 2021

**Type:** Benchmark

## Summary

Scaling BERT-like model inference on CPUs presents practical deployment challenges as transformer architectures proliferate in production environments. Using a reproducible Hydra-based benchmarking framework, inference workloads were evaluated on an Amazon Web Services c5.metal instance featuring an Intel Xeon Platinum 8275 processor with AVX512 and VNNI support. Out-of-the-box evaluations showed PyTorch 1.8.1 delivering lower latency than Google TensorFlow 2.4.0, likely due to PyTorch relying on OpenMP and Intel MKL/oneDNN rather than Eigen. Implementing multiple inference streams by pinning parallel model instances to dedicated CPU cores yielded near-linear throughput scaling. Matching instance core allocations to specific problem sizes also demonstrated that smaller workloads can achieve optimal latency on fewer cores, enabling up to a sixfold reduction in cloud compute costs.

## Context

As transformer architectures grow in production adoption, efficiently deploying and executing BERT-like models at scale on modern CPU architectures remains a technical challenge.

## Approach / What changed

A benchmarking suite built with Facebook AI Research's Hydra was run on an AWS c5.metal bare-metal instance (Intel Xeon Platinum 8275 CPU, 48 cores/96 threads) running Ubuntu 20.04 LTS. The evaluation compared baseline PyTorch 1.8.1 and TensorFlow 2.4.0 configurations, and assessed parallel execution using multiple inference streams where independent model instances are pinned to non-overlapping subsets of CPU cores.

## Takeaways

- Out of the box, PyTorch 1.8.1 demonstrated lower inference latency than TensorFlow 2.4.0 across tested configurations, which may stem from PyTorch using OpenMP and Intel MKL/oneDNN for linear algebra while TensorFlow uses Eigen.
- Running multiple parallel inference streams by allocating dedicated, non-overlapping CPU cores to independent model instances achieved near-linear throughput scalability.
- Smaller workloads such as short sequences and small batch sizes require fewer CPU cores for optimal latency, allowing an 8-core machine at $0.808/hour to achieve a 6x cost reduction over a 48-core machine at $4.848/hour.

**Tags:** [AWS](https://yomu.fyi/topic/aws), [Machine Learning](https://yomu.fyi/topic/machine-learning), [Performance](https://yomu.fyi/topic/performance), [Python](https://yomu.fyi/topic/python), [Scalability](https://yomu.fyi/topic/scalability)

[Read original post](https://huggingface.co/blog/bert-cpu-scaling-part-1)