# Introducing Optimum: The Optimization Toolkit for Transformers at Scale

huggingface.co · Morgan Funtowicz, Ella Charlaix, Michael Benayoun, Jeff Boudier · Sep 14, 2021

**Type:** Announcement

## Summary

Running Transformer models fast and efficiently at production scale presents a significant challenge due to the complex compatibility between model architectures, acceleration techniques like quantization or sparsity, and specific silicon hardware features. Hugging Face introduced Optimum, an open-source library built to abstract hardware acceleration complexity and optimize training and inference on target devices through direct collaborations with hardware partners. Using quantization as a primary example, engineering teams often struggle with eager-mode model graph modifications, kernel compatibility checks, calibration parameter tuning, and acceptable accuracy loss trade-offs. Optimum addresses these challenges by integrating tools such as Intel Neural Compressor, which supports post-training quantization, quantization-aware training, and dynamic quantization driven by user-defined YAML configurations. Through these hardware-tailored integrations, the library aims to democratize production performance and reduce the aggregate energy consumed by machine learning workloads.

## Context

Massive Transformer models require extensive compute power and specialized engineering expertise to run efficiently at scale in production. Achieving optimal performance requires navigating a complex matrix of hardware-specific software tools, silicon operator compatibilities, and model acceleration techniques such as quantization and sparsity.

## Approach / What changed

Hugging Face released Optimum, an open-source optimization toolkit developed with hardware partners to abstract the complexity of hardware-specific acceleration. It integrates low-precision optimization frameworks like Intel Neural Compressor—which supports post-training quantization, quantization-aware training, and dynamic quantization via YAML configuration files—and provides optimized model configurations via the Hugging Face Model Hub.

## Takeaways

- Model acceleration techniques like quantization require matching optimized kernels with silicon operators and neural network computation graphs.
- Quantizing models manually is complex because eager-mode frameworks require editing the model implementation, inserting quantization ops, and calibrating observers.
- Intel Neural Compressor supports post-training quantization, quantization-aware training, and dynamic quantization configured via YAML files hosted locally or on the Hugging Face Hub.

**Tags:** [Machine Learning](https://yomu.fyi/topic/machine-learning), [Open Source](https://yomu.fyi/topic/open-source), [Performance](https://yomu.fyi/topic/performance), [Python](https://yomu.fyi/topic/python)

[Read original post](https://huggingface.co/blog/hardware-partners-program)