Intel and Hugging Face Partner to Democratize Machine Learning Hardware Acceleration

Summary

Intel has joined Hugging Face's Hardware Partner Program to accelerate Transformer training, fine-tuning, and inference on Intel platforms. Large Transformer models introduce latency bottlenecks in production workloads like search and chatbots, where hardware-level optimization typically requires tedious trial and error. To streamline model optimization, the collaboration introduces Optimum Intel, an open-source library integrating the Intel Neural Compressor for automated quantization, pruning, and distillation. A demonstration applies post-training dynamic quantization to a fine-tuned DistilBERT classification model using a CPU-only PyTorch setup. The quantized model converted 38 Linear and 2 Embedding operators to 8-bit integers, reducing evaluation duration by 1.34x while keeping the accuracy drop within a 5% threshold.

Context

Large Transformer models create production challenges for latency-sensitive applications like search and chatbots. Latency optimization is notoriously difficult for machine learning practitioners, requiring extensive trial and error even with deep hardware and framework knowledge.

Approach / What changed

Hugging Face and Intel released Optimum Intel, an open-source library built on Intel Neural Compressor. It integrates automated accuracy-driven compression techniques such as quantization, pruning, and knowledge distillation directly into Transformer workflows for Intel Xeon CPUs and Habana Gaudi accelerators.

Takeaways

• Habana Gaudi accelerators provide up to 40% better price-performance than GPUs for Transformer training.
• Optimum Intel builds on Intel Neural Compressor to automate compression methods like quantization, pruning, and distillation across deep learning frameworks.
• Dynamic quantization of a DistilBERT model with Optimum Intel converted 38 Linear and 2 Embedding operators to INT8, achieving a 1.34x evaluation speedup with an accuracy drop from 0.574 to 0.546.

Related reading

• huggingface.co · Nov 4, 2021

  Scaling up BERT-like model Inference on modern CPU - Part 2

  Scaling transformer inference on modern CPUs requires coordinating hardware capabilities with software optimizations across memory allocation, parallel execution, and mathematical kernel libraries. Intel Ice Lake Xeon processors deliver up to seventy-five percent faster inference across natural language processing tasks compared to Cascade Lake by combining Sunny Cove architecture improvements with dedicated extensions like Intel Extension for PyTorch. Fine-tuning software knobs such as alternative allocators, threading runtimes like Intel OpenMP, and core allocations can involve thousands of possible configuration combinations. Rather than evaluating these combinations through exhaustive brute force, Bayesian optimization via Intel SigOpt achieves comparable latency within roughly twenty trials and reveals experiment-dependent parameter importance. These software tuning strategies enable low-latency production transformer serving without dedicated accelerators.

  Ella Charlaix, Jeff Boudier, Morgan Funtowicz, Michael Benayoun

• huggingface.co · Sep 14, 2021

  Hugging Face and Graphcore partner for IPU-optimized Transformers

  Hugging Face launched its Hardware Partner Program at the 2021 AI Hardware Summit to offer device-optimized models and software integrations. As a founding member of the program, Graphcore collaborated with Hugging Face to optimize Transformer models for its Intelligence Processing Unit (IPU). Graphcore IPUs use a massively parallel MIMD architecture with on-die ultra-high bandwidth memory designed for AI workloads featuring fine-grained parallelism, low precision arithmetic, and sparsity. Through Hugging Face's open-source Optimum toolkit, developers will receive plug-and-play access to certified, hardware-optimized models across vision, speech, translation, and text generation. Graphcore's Poplar SDK integrates with PyTorch, TensorFlow, Docker, and Kubernetes, simplifying the process of porting workloads to IPU-POD datacenter compute systems.