Accelerated Inference with Optimum and Transformers Pipelines

Summary

Transformer-based models are moving into production for large-scale workloads, but default implementations remain slow and resource-intensive compared to traditional machine learning algorithms. Hugging Face Optimum addresses this issue by integrating performance optimization tools and accelerated runtimes like ONNX Runtime directly into standard Transformers pipelines. Developers replace standard model classes with Optimum equivalents to convert models to ONNX, run graph optimizations such as operator fusion, and apply quantization. In an evaluation on an AWS EC2 m5.xlarge CPU instance, optimizing and quantizing a RoBERTa question-answering model cut average latency from 117.61 ms to 64.94 ms while retaining 99.61 percent accuracy. Known limitations include a 2GB remote model size limit, lack of seq2seq support, omitted past key values in causal language models, and no local caching for optimized ONNX models.

Context

Transformer-based architectures deployed to production environments are relatively slow, large, and complex compared to traditional machine learning algorithms.

Approach / What changed

Hugging Face introduced inference and pipeline integration in Optimum 1.2, providing API-compatible ORTModel classes, ORTOptimizer for graph transformations, and ORTQuantizer to convert, optimize, and run models with ONNX Runtime.

Takeaways

• Optimum 1.2 introduces drop-in replacements for Hugging Face AutoModel classes, mapping them to ORTModel classes for execution with ONNX Runtime and Transformers pipelines.
• Graph optimization and dynamic quantization reduced RoBERTa question-answering latency from 117.61 ms to 64.94 ms on an m5.xlarge CPU instance while preserving 99.61 percent accuracy.
• Current limitations include a 2GB limit on remote Hub models, no seq2seq support, lack of past key values in causal generation models, and no local caching for ONNX models.

Related reading

• huggingface.co · Sep 14, 2021

  Introducing Optimum: The Optimization Toolkit for Transformers at Scale

  Running Transformer models fast and efficiently at production scale presents a significant challenge due to the complex compatibility between model architectures, acceleration techniques like quantization or sparsity, and specific silicon hardware features. Hugging Face introduced Optimum, an open-source library built to abstract hardware acceleration complexity and optimize training and inference on target devices through direct collaborations with hardware partners. Using quantization as a primary example, engineering teams often struggle with eager-mode model graph modifications, kernel compatibility checks, calibration parameter tuning, and acceptable accuracy loss trade-offs. Optimum addresses these challenges by integrating tools such as Intel Neural Compressor, which supports post-training quantization, quantization-aware training, and dynamic quantization driven by user-defined YAML configurations. Through these hardware-tailored integrations, the library aims to democratize production performance and reduce the aggregate energy consumed by machine learning workloads.

  Morgan Funtowicz, Ella Charlaix, Michael Benayoun, Jeff Boudier

• huggingface.co · Nov 30, 2021

  Getting Started with Hugging Face Transformers for IPUs with Optimum

  Transformer prediction speed can hinder latency-sensitive workloads like conversational systems and search, while optimizing hardware performance requires significant effort. Hugging Face introduced Optimum, an open-source library that reduces Transformer model latency across diverse hardware targets. Through a partnership with Graphcore, BERT is introduced as the initial IPU-optimized model for parallel AI hardware. Developers configure the Poplar SDK environment and PopTorch inside Graphcloud, install optimum-graphcore, and execute the run_qa.py script with an IPU configuration file. Fine-tuning bert-base-uncased on SQuAD v1.1 using the IPUTrainer class yields an evaluation F1 score of 88.2757 and an exact match score of 80.6623 across 10,784 evaluation samples.