# Accelerated Inference with Optimum and Transformers Pipelines

huggingface.co · Philipp Schmid · May 10, 2022

**Type:** Announcement

## Summary

Transformer-based models are moving into production for large-scale workloads, but default implementations remain slow and resource-intensive compared to traditional machine learning algorithms. Hugging Face Optimum addresses this issue by integrating performance optimization tools and accelerated runtimes like ONNX Runtime directly into standard Transformers pipelines. Developers replace standard model classes with Optimum equivalents to convert models to ONNX, run graph optimizations such as operator fusion, and apply quantization. In an evaluation on an AWS EC2 m5.xlarge CPU instance, optimizing and quantizing a RoBERTa question-answering model cut average latency from 117.61 ms to 64.94 ms while retaining 99.61 percent accuracy. Known limitations include a 2GB remote model size limit, lack of seq2seq support, omitted past key values in causal language models, and no local caching for optimized ONNX models.

## Context

Transformer-based architectures deployed to production environments are relatively slow, large, and complex compared to traditional machine learning algorithms.

## Approach / What changed

Hugging Face introduced inference and pipeline integration in Optimum 1.2, providing API-compatible ORTModel classes, ORTOptimizer for graph transformations, and ORTQuantizer to convert, optimize, and run models with ONNX Runtime.

## Takeaways

- Optimum 1.2 introduces drop-in replacements for Hugging Face AutoModel classes, mapping them to ORTModel classes for execution with ONNX Runtime and Transformers pipelines.
- Graph optimization and dynamic quantization reduced RoBERTa question-answering latency from 117.61 ms to 64.94 ms on an m5.xlarge CPU instance while preserving 99.61 percent accuracy.
- Current limitations include a 2GB limit on remote Hub models, no seq2seq support, lack of past key values in causal generation models, and no local caching for ONNX models.

**Tags:** [Machine Learning](https://yomu.fyi/topic/machine-learning), [Open Source](https://yomu.fyi/topic/open-source), [Performance](https://yomu.fyi/topic/performance), [Python](https://yomu.fyi/topic/python)

[Read original post](https://huggingface.co/blog/optimum-inference)