# Fine-Tune Wav2Vec2 for English ASR in Hugging Face with 🤗 Transformers

huggingface.co · Patrick von Platen · Mar 12, 2021

**Type:** Tutorial

## Summary

Wav2Vec2 is a pretrained Automatic Speech Recognition model that learns speech representations from over 50,000 hours of unlabeled audio using a contrastive objective and masked feature vectors. The model can be fine-tuned end-to-end on labeled speech datasets using Connectionist Temporal Classification without requiring an external language model. To demonstrate this process, a base-sized checkpoint is fine-tuned on the Timit dataset, which contains five hours of training data. Data preparation involves normalizing transcriptions to lowercase, removing special punctuation characters, and configuring both a specialized feature extractor and tokenizer. Evaluating the fine-tuned acoustic model without padding on the Timit test dataset yields a word error rate of 22.1%.

## Context

Pretrained on more than 50,000 hours of unlabeled speech, Wav2Vec2 can achieve competitive speech recognition performance with very little labeled data. However, practitioners need a streamlined method to fine-tune pretrained checkpoints on custom English speech datasets as standalone end-to-end acoustic models.

## Approach / What changed

A base-sized Wav2Vec2 checkpoint is fine-tuned on five hours of Timit speech data using Connectionist Temporal Classification (CTC) without an auxiliary language model. Transcriptions are cleaned by stripping special punctuation and converting text to lowercase to build a character vocabulary. A linear classification layer is placed on top of the transformer representations, and inference is run with a batch size of one to prevent padding artifacts from degrading the word error rate.

## Takeaways

- Wav2Vec2 pretrained checkpoints can be fine-tuned directly for speech-to-text tasks via Connectionist Temporal Classification (CTC) without relying on a language model.
- Punctuation marks are removed during text preprocessing because characters like periods lack distinct acoustic signatures, which makes classification without a language model difficult.
- Evaluating test audio with a batch size of one avoids input padding, which yields a better word error rate because padded inputs alter output representations.

**Tags:** [Machine Learning](https://yomu.fyi/topic/machine-learning), [Open Source](https://yomu.fyi/topic/open-source), [Python](https://yomu.fyi/topic/python)

[Read original post](https://huggingface.co/blog/fine-tune-wav2vec2-english)