Getting Started with Hugging Face Transformers for IPUs with Optimum

Summary

Transformer prediction speed can hinder latency-sensitive workloads like conversational systems and search, while optimizing hardware performance requires significant effort. Hugging Face introduced Optimum, an open-source library that reduces Transformer model latency across diverse hardware targets. Through a partnership with Graphcore, BERT is introduced as the initial IPU-optimized model for parallel AI hardware. Developers configure the Poplar SDK environment and PopTorch inside Graphcloud, install optimum-graphcore, and execute the run_qa.py script with an IPU configuration file. Fine-tuning bert-base-uncased on SQuAD v1.1 using the IPUTrainer class yields an evaluation F1 score of 88.2757 and an exact match score of 80.6623 across 10,784 evaluation samples.

Context

Large Transformer model prediction speeds make them impractical for latency-sensitive applications like search or conversational systems, and optimizing their real-world hardware performance requires specialized skills and effort beyond many organizations' reach.

Approach / What changed

Graphcore and Hugging Face integrated IPU acceleration into the open-source Optimum library, enabling developers to run BERT on Graphcore IPU-POD16 systems via Poplar SDK, PopTorch, and the optimum-graphcore package using the IPUTrainer class.

Takeaways

• Optimum serves as the interface between Hugging Face Transformers and Graphcore IPUs, enabling training and fine-tuning with minimal code modifications via PopTorch and IPUTrainer.
• The Optimum Graphcore question-answering fine-tuning script run_qa.py requires a fast tokenizer backed by the Hugging Face Tokenizers library and an ipu_config.json configuration file.
• Fine-tuning bert-base-uncased on SQuAD v1.1 for 3 epochs on the IPU setup completed in 368.40 seconds, achieving an evaluation F1 score of 88.2757 and exact match of 80.6623.

Related reading

• huggingface.co · May 26, 2022

  Graphcore and Hugging Face Launch New Lineup of IPU-Ready Transformers

  Graphcore and Hugging Face expanded the range of machine learning modalities and tasks available in Hugging Face Optimum. Developers can now access ten transformer models optimized for Graphcore IPUs across natural language processing, speech, and computer vision. The available architectures include BERT, ViT, GPT-2, RoBERTa, DeBERTa, BART, LXMERT, T5, HuBERT, and Wav2Vec2, complete with IPU configuration files and ready-to-use pre-trained or fine-tuned weights. The integration supports the Bow IPU processor, which uses 3D Wafer-on-Wafer stacking to achieve up to 350 teraFLOPS of AI compute. Optimum also integrates with the Poplar SDK 2.5, enabling compatibility with frameworks such as PyTorch, TensorFlow, Docker, and Kubernetes.

  Sally Doherty

• huggingface.co · Sep 14, 2021

  Hugging Face and Graphcore partner for IPU-optimized Transformers

  Hugging Face launched its Hardware Partner Program at the 2021 AI Hardware Summit to offer device-optimized models and software integrations. As a founding member of the program, Graphcore collaborated with Hugging Face to optimize Transformer models for its Intelligence Processing Unit (IPU). Graphcore IPUs use a massively parallel MIMD architecture with on-die ultra-high bandwidth memory designed for AI workloads featuring fine-grained parallelism, low precision arithmetic, and sparsity. Through Hugging Face's open-source Optimum toolkit, developers will receive plug-and-play access to certified, hardware-optimized models across vision, speech, translation, and text generation. Graphcore's Poplar SDK integrates with PyTorch, TensorFlow, Docker, and Kubernetes, simplifying the process of porting workloads to IPU-POD datacenter compute systems.