Introducing Optimum: The Optimization Toolkit for Transformers at Scale

Summary

Running Transformer models fast and efficiently at production scale presents a significant challenge due to the complex compatibility between model architectures, acceleration techniques like quantization or sparsity, and specific silicon hardware features. Hugging Face introduced Optimum, an open-source library built to abstract hardware acceleration complexity and optimize training and inference on target devices through direct collaborations with hardware partners. Using quantization as a primary example, engineering teams often struggle with eager-mode model graph modifications, kernel compatibility checks, calibration parameter tuning, and acceptable accuracy loss trade-offs. Optimum addresses these challenges by integrating tools such as Intel Neural Compressor, which supports post-training quantization, quantization-aware training, and dynamic quantization driven by user-defined YAML configurations. Through these hardware-tailored integrations, the library aims to democratize production performance and reduce the aggregate energy consumed by machine learning workloads.

Context

Massive Transformer models require extensive compute power and specialized engineering expertise to run efficiently at scale in production. Achieving optimal performance requires navigating a complex matrix of hardware-specific software tools, silicon operator compatibilities, and model acceleration techniques such as quantization and sparsity.

Approach / What changed

Hugging Face released Optimum, an open-source optimization toolkit developed with hardware partners to abstract the complexity of hardware-specific acceleration. It integrates low-precision optimization frameworks like Intel Neural Compressor—which supports post-training quantization, quantization-aware training, and dynamic quantization via YAML configuration files—and provides optimized model configurations via the Hugging Face Model Hub.

Takeaways

• Model acceleration techniques like quantization require matching optimized kernels with silicon operators and neural network computation graphs.
• Quantizing models manually is complex because eager-mode frameworks require editing the model implementation, inserting quantization ops, and calibrating observers.
• Intel Neural Compressor supports post-training quantization, quantization-aware training, and dynamic quantization configured via YAML files hosted locally or on the Hugging Face Hub.

Related reading

• huggingface.co · Jun 15, 2022

  Intel and Hugging Face Partner to Democratize Machine Learning Hardware Acceleration

  Intel has joined Hugging Face's Hardware Partner Program to accelerate Transformer training, fine-tuning, and inference on Intel platforms. Large Transformer models introduce latency bottlenecks in production workloads like search and chatbots, where hardware-level optimization typically requires tedious trial and error. To streamline model optimization, the collaboration introduces Optimum Intel, an open-source library integrating the Intel Neural Compressor for automated quantization, pruning, and distillation. A demonstration applies post-training dynamic quantization to a fine-tuned DistilBERT classification model using a CPU-only PyTorch setup. The quantized model converted 38 Linear and 2 Embedding operators to 8-bit integers, reducing evaluation duration by 1.34x while keeping the accuracy drop within a 5% threshold.

  Julien Simon

• huggingface.co · Jan 18, 2021

  How we sped up transformer inference 100x for 🤗 API customers

  Deploying large transformer models in production poses severe latency and scaling hurdles for machine learning engineers. Hugging Face achieved a cumulative 100x inference acceleration on its Accelerated Inference API by pairing library-level algorithmic adjustments with low-level hardware compilation. The initial 10x improvement reduces forward-pass computations, such as restricting attention calculations to the latest token in GPT generation tasks, while utilizing Rust-based tokenizers and caching mechanisms. The subsequent 10x speedup requires custom static graph compilation targeted to selected CPU or GPU hardware profiles, applying unused flow removal, instruction-specific layer fusion, and precision quantization via ONNX Runtime. These coordinated techniques enable real-time prediction latency across varied model sizes and demand profiles without sacrificing task accuracy.