How we sped up transformer inference 100x for 🤗 API customers

Summary

Deploying large transformer models in production poses severe latency and scaling hurdles for machine learning engineers. Hugging Face achieved a cumulative 100x inference acceleration on its Accelerated Inference API by pairing library-level algorithmic adjustments with low-level hardware compilation. The initial 10x improvement reduces forward-pass computations, such as restricting attention calculations to the latest token in GPT generation tasks, while utilizing Rust-based tokenizers and caching mechanisms. The subsequent 10x speedup requires custom static graph compilation targeted to selected CPU or GPU hardware profiles, applying unused flow removal, instruction-specific layer fusion, and precision quantization via ONNX Runtime. These coordinated techniques enable real-time prediction latency across varied model sizes and demand profiles without sacrificing task accuracy.

Context

Deploying large Transformer models into production with maximum performance and scalable architectures presents major engineering challenges for machine learning engineers. With models expanding from 110M parameters to 175B parameters, reaching real-time latency for consumer applications requires massive inference speedups.

Approach / What changed

Hugging Face combined high-level library optimizations with low-level, hardware-targeted compilation to achieve a 100x inference speedup on their Accelerated Inference API. The initial 10x gain optimizes model pipelines (such as evaluating only the last token attention in GPT architectures) alongside Rust-based tokenization and caching. The final 10x gain comes from custom hardware compilation, including CPU graph optimization, layer fusion, and quantization using ONNX Runtime.

Takeaways

• Tokenization bottlenecks can be mitigated by combining Rust-based tokenizer implementations with smart caching to reduce overall latency by up to 10x.
• Architectural optimizations, such as focusing attention matrix computations strictly on the newest token during GPT text generation passes, reduce forward-pass compute.
• Hardware-specific static graph compilation on CPUs achieves further acceleration via unused flow elimination, CPU-instruction layer fusion, and targeted operation quantization.

Related reading

• huggingface.co · Oct 20, 2021

  The Age of Machine Learning As Code Has Arrived

  Recent findings from the 2021 State of AI Report and Kaggle State of Machine Learning and Data Science Survey indicate that machine learning is expanding into critical infrastructure while Transformers become general-purpose architectures across text, vision, and audio. In response, organizations face questions about scaling infrastructure, team composition, and engineering maturity. Rather than treating machine learning as isolated sandbox experiments or hiring solely data scientists, teams benefit from adopting established software engineering and DevOps principles like versioning, testing, automation, and continuous deployment. Furthermore, the rise of pre-trained Transformer architectures enables practitioners to fine-tune existing off-the-shelf models rather than training from scratch, reducing compute costs and training duration. Tools from platforms such as Hugging Face streamline model deployment, latency optimization, and infrastructure abstraction.

  Julien Simon

• huggingface.co · Sep 14, 2021

  Introducing Optimum: The Optimization Toolkit for Transformers at Scale

  Running Transformer models fast and efficiently at production scale presents a significant challenge due to the complex compatibility between model architectures, acceleration techniques like quantization or sparsity, and specific silicon hardware features. Hugging Face introduced Optimum, an open-source library built to abstract hardware acceleration complexity and optimize training and inference on target devices through direct collaborations with hardware partners. Using quantization as a primary example, engineering teams often struggle with eager-mode model graph modifications, kernel compatibility checks, calibration parameter tuning, and acceptable accuracy loss trade-offs. Optimum addresses these challenges by integrating tools such as Intel Neural Compressor, which supports post-training quantization, quantization-aware training, and dynamic quantization driven by user-defined YAML configurations. Through these hardware-tailored integrations, the library aims to democratize production performance and reduce the aggregate energy consumed by machine learning workloads.