Scaling-up BERT Inference on CPU (Part 1)

Summary

Scaling BERT-like model inference on CPUs presents practical deployment challenges as transformer architectures proliferate in production environments. Using a reproducible Hydra-based benchmarking framework, inference workloads were evaluated on an Amazon Web Services c5.metal instance featuring an Intel Xeon Platinum 8275 processor with AVX512 and VNNI support. Out-of-the-box evaluations showed PyTorch 1.8.1 delivering lower latency than Google TensorFlow 2.4.0, likely due to PyTorch relying on OpenMP and Intel MKL/oneDNN rather than Eigen. Implementing multiple inference streams by pinning parallel model instances to dedicated CPU cores yielded near-linear throughput scaling. Matching instance core allocations to specific problem sizes also demonstrated that smaller workloads can achieve optimal latency on fewer cores, enabling up to a sixfold reduction in cloud compute costs.

Takeaways

1. 1

   Out of the box, PyTorch 1.8.1 demonstrated lower inference latency than TensorFlow 2.4.0 across tested configurations, which may stem from PyTorch using OpenMP and Intel MKL/oneDNN for linear algebra while TensorFlow uses Eigen.

2. 2

   Running multiple parallel inference streams by allocating dedicated, non-overlapping CPU cores to independent model instances achieved near-linear throughput scalability.

3. 3

   Smaller workloads such as short sequences and small batch sizes require fewer CPU cores for optimal latency, allowing an 8-core machine at $0.808/hour to achieve a 6x cost reduction over a 48-core machine at $4.848/hour.