Intel and Hugging Face Partner to Democratize Machine Learning Hardware Acceleration

Summary

Intel has joined Hugging Face's Hardware Partner Program to accelerate Transformer training, fine-tuning, and inference on Intel platforms. Large Transformer models introduce latency bottlenecks in production workloads like search and chatbots, where hardware-level optimization typically requires tedious trial and error. To streamline model optimization, the collaboration introduces Optimum Intel, an open-source library integrating the Intel Neural Compressor for automated quantization, pruning, and distillation. A demonstration applies post-training dynamic quantization to a fine-tuned DistilBERT classification model using a CPU-only PyTorch setup. The quantized model converted 38 Linear and 2 Embedding operators to 8-bit integers, reducing evaluation duration by 1.34x while keeping the accuracy drop within a 5% threshold.

Takeaways

1. 1

   Habana Gaudi accelerators provide up to 40% better price-performance than GPUs for Transformer training.

2. 2

   Optimum Intel builds on Intel Neural Compressor to automate compression methods like quantization, pruning, and distillation across deep learning frameworks.

3. 3

   Dynamic quantization of a DistilBERT model with Optimum Intel converted 38 Linear and 2 Embedding operators to INT8, achieving a 1.34x evaluation speedup with an accuracy drop from 0.574 to 0.546.