Making automatic speech recognition work on large files with Wav2Vec2 in 🤗 Transformers

Summary

Transformer-based automatic speech recognition models like Wav2Vec2 crash on long audio inputs because the O(n²) attention mechanism rapidly exhausts GPU memory. Simple audio chunking avoids out-of-memory errors but causes poor transcription accuracy at chunk boundaries due to a lack of surrounding context. To resolve this, Hugging Face Transformers leverages the Connectionist Temporal Classification architecture to process overlapping audio chunks with configurable strides. The pipeline discards low-quality logits at chunk edges and chains the remaining central predictions together to reconstruct a seamless transcript. This striding technique operates out of the box with language model-augmented pipelines and adapts directly to low-latency live audio streaming.

Context

Running Wav2Vec2 directly on long audio files causes out-of-memory crashes because transformer attention complexity scales quadratically with sequence length. While naive chunking prevents crashes, lack of context around cut boundaries degrades recognition quality, and silence- or voice-detection workarounds are unreliable.

Approach / What changed

Hugging Face Transformers leverages the Connectionist Temporal Classification (CTC) architecture to process overlapping chunks using configurable stride parameters (chunk_length_s and stride_length_s). The pipeline generates logits for overlapping segments, discards the boundary logits where accuracy is low, and chains the central logits together before decoding.

Takeaways

• Transformer attention complexity scales quadratically with sequence length, causing unchunked Wav2Vec2 speech recognition inference to crash with out-of-memory errors on long audio files.
• Connectionist Temporal Classification (CTC) maps each audio frame to a letter prediction logit, allowing pipelines to drop edge logits from overlapping chunks and concatenate the high-quality central outputs.
• The chunking and striding method applies directly to language model-augmented Wav2Vec2 checkpoints and enables live streaming inference by processing overlapping incoming audio frames.

Related reading

• huggingface.co · Jan 12, 2022

  Boosting Wav2Vec2 with n-grams in 🤗 Transformers

  Wav2Vec2 models fine-tuned with Connectionist Temporal Classification transcribe speech without external language models, but decoding can still suffer from spelling inaccuracies. Hugging Face Transformers addressed this by integrating Kensho Technologies' pyctcdecode library to support decoding with n-gram language models. Instead of decoding using simple argmax operations over logits, the Wav2Vec2ProcessorWithLM class feeds full probability matrices into beam search guided by KenLM n-gram probabilities. KenLM's build_binary utility compresses language model ARPA files into binary formats, reducing file size by more than half for faster loading and hub deployment. In Swedish xls-r-300m-sv benchmarks on Common Voice 7, this 5-gram boosted decoding setup achieved an 18.85% word error rate, delivering an approximate 30% relative performance gain.

  Patrick von Platen

• huggingface.co · Mar 12, 2021

  Fine-Tune Wav2Vec2 for English ASR in Hugging Face with 🤗 Transformers

  Wav2Vec2 is a pretrained Automatic Speech Recognition model that learns speech representations from over 50,000 hours of unlabeled audio using a contrastive objective and masked feature vectors. The model can be fine-tuned end-to-end on labeled speech datasets using Connectionist Temporal Classification without requiring an external language model. To demonstrate this process, a base-sized checkpoint is fine-tuned on the Timit dataset, which contains five hours of training data. Data preparation involves normalizing transcriptions to lowercase, removing special punctuation characters, and configuring both a specialized feature extractor and tokenizer. Evaluating the fine-tuned acoustic model without padding on the Timit test dataset yields a word error rate of 22.1%.