Accelerating PyTorch distributed fine-tuning with Intel technologies

Summary

Transfer learning makes CPU-based clusters a viable, cost-effective option for shorter deep learning fine-tuning jobs. To accelerate PyTorch workloads, clusters can leverage Intel Ice Lake architecture featuring AVX-512 and Vector Neural Network Instructions alongside specialized software optimizations. Implementing the Intel Extension for PyTorch and the oneAPI Collective Communications Library (oneCCL) addresses communication bottlenecks across distributed nodes. In demonstrated tests fine-tuning transformer models on GLUE tasks across Amazon EC2 c6i.16xlarge instances, two-node jobs delivered roughly a 1.7x speedup, while four-node setups achieved a 3x speedup. These performance libraries and multi-node configurations maintain consistent scaling across both smaller datasets like MRPC and larger benchmarks like QQP.

Context

State-of-the-art deep learning models often take a long time to train, but transfer learning involves shorter fine-tuning jobs where CPU clusters can control time and cost compared to traditional GPU training.

Approach / What changed

Provision a multi-node cluster of Intel Ice Lake CPU instances, install Intel oneAPI toolkits alongside the Intel Extension for PyTorch and oneCCL, adapt training scripts for distributed CCL execution, and run distributed jobs using mpirun.

Takeaways

• Intel Ice Lake and Cascade Lake CPUs accelerate deep learning operations via hardware features like AVX-512 and Vector Neural Network Instructions (VNNI).
• The Intel oneAPI Collective Communications Library (oneCCL) provides optimized implementations of deep learning communication patterns like all-reduce to reduce networking bottlenecks.
• Scaling distilbert-base-uncased fine-tuning across two and four Intel Xeon nodes produced consistent speedups of approximately 1.7x and 3x on both MRPC and QQP GLUE benchmarks.

Related reading

• huggingface.co · Jan 13, 2022

  Case Study: Millisecond Latency using Hugging Face Infinity and modern CPUs

  Deploying large Transformer models at scale often faces severe prediction latency bottlenecks, driving up infrastructure costs and limiting real-time production use cases. To address this challenge, Hugging Face evaluated Infinity, a containerized hardware-optimized inference solution paired with the Infinity Multiverse model optimization service. Testing covered 192 configurations on Amazon EC2 C6i instances powered by 3rd generation Intel Xeon Scalable processors across varying CPU cores, sequence lengths, and batch sizes. The benchmarks demonstrated that an Ice Lake-optimized DistilBERT container achieved up to 800% higher throughput than vanilla Transformers and delivered 1 to 4 millisecond end-to-end latencies for sequence lengths up to 64 tokens. Although Infinity was later discontinued in favor of Inference Endpoints and Optimum libraries, the results demonstrated substantial efficiency gains on modern CPU hardware.

  Philipp Schmid, Jeff Boudier, Morgan Funtowicz

• huggingface.co · Nov 4, 2021

  Scaling up BERT-like model Inference on modern CPU - Part 2

  Scaling transformer inference on modern CPUs requires coordinating hardware capabilities with software optimizations across memory allocation, parallel execution, and mathematical kernel libraries. Intel Ice Lake Xeon processors deliver up to seventy-five percent faster inference across natural language processing tasks compared to Cascade Lake by combining Sunny Cove architecture improvements with dedicated extensions like Intel Extension for PyTorch. Fine-tuning software knobs such as alternative allocators, threading runtimes like Intel OpenMP, and core allocations can involve thousands of possible configuration combinations. Rather than evaluating these combinations through exhaustive brute force, Bayesian optimization via Intel SigOpt achieves comparable latency within roughly twenty trials and reveals experiment-dependent parameter importance. These software tuning strategies enable low-latency production transformer serving without dedicated accelerators.