Fine-Tune XLSR-Wav2Vec2 for low-resource ASR with 🤗 Transformers

Summary

Fine-tuning the multilingual XLS-R speech model adapts pretrained cross-lingual audio representations to automatic speech recognition tasks with limited labeled data. The demonstration configures a Wav2Vec2-XLS-R-300M checkpoint using Connectionist Temporal Classification on roughly four hours of validated Turkish audio from Common Voice. Building the pipeline requires pairing a Wav2Vec2FeatureExtractor for audio signals with a custom Wav2Vec2CTCTokenizer derived from dataset transcriptions. Training with Hugging Face Trainer over 3,200 steps decreases the validation word error rate from 0.7000 down to 0.3195. While the resulting transcription demonstrates recognizable phonetic alignment, output quality can be further improved by extending training schedules, refining preprocessing, and adding language model decoding.

Context

Automatic speech recognition for low-resource languages faces challenges due to limited labeled training data, necessitating cross-lingual models pretrained on multilingual speech.

Approach / What changed

Fine-tuning a Wav2Vec2-XLS-R-300M model on approximately four hours of Turkish Common Voice audio by appending a linear output layer and optimizing via Connectionist Temporal Classification.

Takeaways

• XLS-R learns contextualized speech representations across 128 languages using self-supervised masked feature prediction, with pretrained checkpoints scaling from 300M to 2B parameters.
• For ASR downstream tasks, a linear layer mapping context representations to vocabulary tokens is added atop the pretrained transformer and trained with Connectionist Temporal Classification.
• To handle GPU out-of-memory errors during training, reduce per_device_train_batch_size to 8 or less and increase gradient_accumulation.

Related reading

• huggingface.co · Mar 12, 2021

  Fine-Tune Wav2Vec2 for English ASR in Hugging Face with 🤗 Transformers

  Wav2Vec2 is a pretrained Automatic Speech Recognition model that learns speech representations from over 50,000 hours of unlabeled audio using a contrastive objective and masked feature vectors. The model can be fine-tuned end-to-end on labeled speech datasets using Connectionist Temporal Classification without requiring an external language model. To demonstrate this process, a base-sized checkpoint is fine-tuned on the Timit dataset, which contains five hours of training data. Data preparation involves normalizing transcriptions to lowercase, removing special punctuation characters, and configuring both a specialized feature extractor and tokenizer. Evaluating the fine-tuned acoustic model without padding on the Timit test dataset yields a word error rate of 22.1%.

  Patrick von Platen

• huggingface.co · Jan 12, 2022

  Boosting Wav2Vec2 with n-grams in 🤗 Transformers

  Wav2Vec2 models fine-tuned with Connectionist Temporal Classification transcribe speech without external language models, but decoding can still suffer from spelling inaccuracies. Hugging Face Transformers addressed this by integrating Kensho Technologies' pyctcdecode library to support decoding with n-gram language models. Instead of decoding using simple argmax operations over logits, the Wav2Vec2ProcessorWithLM class feeds full probability matrices into beam search guided by KenLM n-gram probabilities. KenLM's build_binary utility compresses language model ARPA files into binary formats, reducing file size by more than half for faster loading and hub deployment. In Swedish xls-r-300m-sv benchmarks on Common Voice 7, this 5-gram boosted decoding setup achieved an 18.85% word error rate, delivering an approximate 30% relative performance gain.