Code-Switching Detection via Language-Selective Dimensions in Speech Foundation Models

Summary

Code-switching, the alternation between languages in a single utterance, is common in multilingual speech but underrepresented in training corpora for speech models. To address this, a lightweight method detects code-switching directly from speech without requiring multilingual transcriptions. The approach leverages language-selective dimensions in pretrained speech encoders by identifying hidden-state dimensions selective to each language from monolingual recordings and aggregating frame-level activations to predict code-switching. On the CS-FLEURS dataset covering 14 language pairs, the Whisper-small encoder achieves a mean F1 of 0.798, outperforming a CNN baseline (0.776) and a HuBERT linear baseline (0.751). These results show that pretrained speech encoders inherently encode the information necessary for code-switching detection, offering a scalable tool for building more representative multilingual corpora.

Takeaways

1. 1

   Code-switching detection can be performed directly from speech without text transcription by using language-selective dimensions in pretrained encoders.

2. 2

   On CS-FLEURS across 14 language pairs, Whisper-small achieves mean F1 0.798, outperforming CNN (0.776) and HuBERT linear (0.751) baselines.

3. 3

   The lightweight method requires only a small code-switching set to tune a decision rule, enabling scalable curation of diverse multilingual training data.

FAQ