# Fine-Tune XLSR-Wav2Vec2 for low-resource ASR with 🤗 Transformers

huggingface.co · Patrick von Platen · Nov 15, 2021

**Type:** Tutorial

## Summary

Fine-tuning the multilingual XLS-R speech model adapts pretrained cross-lingual audio representations to automatic speech recognition tasks with limited labeled data. The demonstration configures a Wav2Vec2-XLS-R-300M checkpoint using Connectionist Temporal Classification on roughly four hours of validated Turkish audio from Common Voice. Building the pipeline requires pairing a Wav2Vec2FeatureExtractor for audio signals with a custom Wav2Vec2CTCTokenizer derived from dataset transcriptions. Training with Hugging Face Trainer over 3,200 steps decreases the validation word error rate from 0.7000 down to 0.3195. While the resulting transcription demonstrates recognizable phonetic alignment, output quality can be further improved by extending training schedules, refining preprocessing, and adding language model decoding.

## Context

Automatic speech recognition for low-resource languages faces challenges due to limited labeled training data, necessitating cross-lingual models pretrained on multilingual speech.

## Approach / What changed

Fine-tuning a Wav2Vec2-XLS-R-300M model on approximately four hours of Turkish Common Voice audio by appending a linear output layer and optimizing via Connectionist Temporal Classification.

## Takeaways

- XLS-R learns contextualized speech representations across 128 languages using self-supervised masked feature prediction, with pretrained checkpoints scaling from 300M to 2B parameters.
- For ASR downstream tasks, a linear layer mapping context representations to vocabulary tokens is added atop the pretrained transformer and trained with Connectionist Temporal Classification.
- To handle GPU out-of-memory errors during training, reduce per\_device\_train\_batch\_size to 8 or less and increase gradient\_accumulation.

**Tags:** [Machine Learning](https://yomu.fyi/topic/machine-learning), [Open Source](https://yomu.fyi/topic/open-source), [Python](https://yomu.fyi/topic/python)

[Read original post](https://huggingface.co/blog/fine-tune-xlsr-wav2vec2)