# How we sped up transformer inference 100x for 🤗 API customers

huggingface.co · Nicolas Patry · Jan 18, 2021

**Type:** Problem & solution

## Summary

Deploying large transformer models in production poses severe latency and scaling hurdles for machine learning engineers. Hugging Face achieved a cumulative 100x inference acceleration on its Accelerated Inference API by pairing library-level algorithmic adjustments with low-level hardware compilation. The initial 10x improvement reduces forward-pass computations, such as restricting attention calculations to the latest token in GPT generation tasks, while utilizing Rust-based tokenizers and caching mechanisms. The subsequent 10x speedup requires custom static graph compilation targeted to selected CPU or GPU hardware profiles, applying unused flow removal, instruction-specific layer fusion, and precision quantization via ONNX Runtime. These coordinated techniques enable real-time prediction latency across varied model sizes and demand profiles without sacrificing task accuracy.

## Context

Deploying large Transformer models into production with maximum performance and scalable architectures presents major engineering challenges for machine learning engineers. With models expanding from 110M parameters to 175B parameters, reaching real-time latency for consumer applications requires massive inference speedups.

## Approach / What changed

Hugging Face combined high-level library optimizations with low-level, hardware-targeted compilation to achieve a 100x inference speedup on their Accelerated Inference API. The initial 10x gain optimizes model pipelines (such as evaluating only the last token attention in GPT architectures) alongside Rust-based tokenization and caching. The final 10x gain comes from custom hardware compilation, including CPU graph optimization, layer fusion, and quantization using ONNX Runtime.

## Takeaways

- Tokenization bottlenecks can be mitigated by combining Rust-based tokenizer implementations with smart caching to reduce overall latency by up to 10x.
- Architectural optimizations, such as focusing attention matrix computations strictly on the newest token during GPT text generation passes, reduce forward-pass compute.
- Hardware-specific static graph compilation on CPUs achieves further acceleration via unused flow elimination, CPU-instruction layer fusion, and targeted operation quantization.

**Tags:** [Caching](https://yomu.fyi/topic/caching), [Machine Learning](https://yomu.fyi/topic/machine-learning), [Open Source](https://yomu.fyi/topic/open-source), [Performance](https://yomu.fyi/topic/performance), [Rust](https://yomu.fyi/topic/rust)

[Read original post](https://huggingface.co/blog/accelerated-inference)