Accelerated Inference with Optimum and Transformers Pipelines

Summary

Transformer-based models are moving into production for large-scale workloads, but default implementations remain slow and resource-intensive compared to traditional machine learning algorithms. Hugging Face Optimum addresses this issue by integrating performance optimization tools and accelerated runtimes like ONNX Runtime directly into standard Transformers pipelines. Developers replace standard model classes with Optimum equivalents to convert models to ONNX, run graph optimizations such as operator fusion, and apply quantization. In an evaluation on an AWS EC2 m5.xlarge CPU instance, optimizing and quantizing a RoBERTa question-answering model cut average latency from 117.61 ms to 64.94 ms while retaining 99.61 percent accuracy. Known limitations include a 2GB remote model size limit, lack of seq2seq support, omitted past key values in causal language models, and no local caching for optimized ONNX models.

Takeaways

1. 1

   Optimum 1.2 introduces drop-in replacements for Hugging Face AutoModel classes, mapping them to ORTModel classes for execution with ONNX Runtime and Transformers pipelines.

2. 2

   Graph optimization and dynamic quantization reduced RoBERTa question-answering latency from 117.61 ms to 64.94 ms on an m5.xlarge CPU instance while preserving 99.61 percent accuracy.

3. 3

   Current limitations include a 2GB limit on remote Hub models, no seq2seq support, lack of past key values in causal generation models, and no local caching for ONNX models.