# Boosting Wav2Vec2 with n-grams in 🤗 Transformers

huggingface.co · Patrick von Platen · Jan 12, 2022

**Type:** Tutorial

## Summary

Wav2Vec2 models fine-tuned with Connectionist Temporal Classification transcribe speech without external language models, but decoding can still suffer from spelling inaccuracies. Hugging Face Transformers addressed this by integrating Kensho Technologies' pyctcdecode library to support decoding with n-gram language models. Instead of decoding using simple argmax operations over logits, the Wav2Vec2ProcessorWithLM class feeds full probability matrices into beam search guided by KenLM n-gram probabilities. KenLM's build\_binary utility compresses language model ARPA files into binary formats, reducing file size by more than half for faster loading and hub deployment. In Swedish xls-r-300m-sv benchmarks on Common Voice 7, this 5-gram boosted decoding setup achieved an 18.85% word error rate, delivering an approximate 30% relative performance gain.

## Context

Pre-trained Wav2Vec2 models can perform speech recognition without external language models due to transformer contextualization and CTC alignment, but decoding without a language model still produces spelling errors. Additionally, Hugging Face Transformers previously lacked a simple user interface to decode audio using fine-tuned Wav2Vec2 checkpoints combined with a language model.

## Approach / What changed

Hugging Face Transformers integrated Kensho Technologies' pyctcdecode library and introduced Wav2Vec2ProcessorWithLM to combine fine-tuned checkpoints with KenLM n-gram language models. The workflow passes raw model output logits directly into pyctcdecode to perform beam search guided by n-gram probabilities, and utilizes KenLM's build\_binary tool to convert large ARPA language model files into optimized binary formats.

## Takeaways

- Decoding with Wav2Vec2ProcessorWithLM uses full output logits and beam search weighted by n-gram probabilities rather than taking greedy argmax selections.
- Converting KenLM .arpa language model files to binary format via build\_binary reduced a 5-gram LM file size from over 4 GB to 1.8 GB.
- Adding a 5-gram language model to the xls-r-300m-sv checkpoint achieved an 18.85% WER on the Common Voice 7 Swedish test set, yielding an approximate 30% relative improvement.

**Tags:** [Machine Learning](https://yomu.fyi/topic/machine-learning), [Open Source](https://yomu.fyi/topic/open-source), [Performance](https://yomu.fyi/topic/performance), [Python](https://yomu.fyi/topic/python)

[Read original post](https://huggingface.co/blog/wav2vec2-with-ngram)