# Intel and Hugging Face Partner to Democratize Machine Learning Hardware Acceleration

huggingface.co · Julien Simon · Jun 15, 2022

**Type:** Announcement

## Summary

Intel has joined Hugging Face's Hardware Partner Program to accelerate Transformer training, fine-tuning, and inference on Intel platforms. Large Transformer models introduce latency bottlenecks in production workloads like search and chatbots, where hardware-level optimization typically requires tedious trial and error. To streamline model optimization, the collaboration introduces Optimum Intel, an open-source library integrating the Intel Neural Compressor for automated quantization, pruning, and distillation. A demonstration applies post-training dynamic quantization to a fine-tuned DistilBERT classification model using a CPU-only PyTorch setup. The quantized model converted 38 Linear and 2 Embedding operators to 8-bit integers, reducing evaluation duration by 1.34x while keeping the accuracy drop within a 5% threshold.

## Context

Large Transformer models create production challenges for latency-sensitive applications like search and chatbots. Latency optimization is notoriously difficult for machine learning practitioners, requiring extensive trial and error even with deep hardware and framework knowledge.

## Approach / What changed

Hugging Face and Intel released Optimum Intel, an open-source library built on Intel Neural Compressor. It integrates automated accuracy-driven compression techniques such as quantization, pruning, and knowledge distillation directly into Transformer workflows for Intel Xeon CPUs and Habana Gaudi accelerators.

## Takeaways

- Habana Gaudi accelerators provide up to 40% better price-performance than GPUs for Transformer training.
- Optimum Intel builds on Intel Neural Compressor to automate compression methods like quantization, pruning, and distillation across deep learning frameworks.
- Dynamic quantization of a DistilBERT model with Optimum Intel converted 38 Linear and 2 Embedding operators to INT8, achieving a 1.34x evaluation speedup with an accuracy drop from 0.574 to 0.546.

**Tags:** [Machine Learning](https://yomu.fyi/topic/machine-learning), [Open Source](https://yomu.fyi/topic/open-source), [Performance](https://yomu.fyi/topic/performance), [Python](https://yomu.fyi/topic/python)

[Read original post](https://huggingface.co/blog/intel)