Language Discrimination Improves Linguistic Learning in Multilingual Speech Models

Summary

Multilingual self-supervised speech models underperform monolingual models under a matched total pretraining data budget, but strengthening language discrimination during pretraining reduces this gap while preserving cross-language sharing. In a controlled English/French HuBERT setting, two interventions—an auxiliary language classifier and per-language k-means targets—improve language discrimination. Continuous-feature phone discrimination error (phone-ABX) decreased from 11.6% in the bilingual baseline to 10.4%, approaching the monolingual 10.8%. Lexical performance (sWUGGY) rose from 52.1% to 56.7% (monolingual: 58.5%), and prosodic performance (ProsAudit lexical subtask) from 68.9% to 72.9% (monolingual: 72.6%). The strongest gains occur when language discrimination is introduced in the first HuBERT iteration; later or repeated interventions yield smaller improvements and increase language-wise segregation, supporting a causal role for language discrimination in reducing the cost of multilingual learning.

Takeaways

1. 1

   Strengthening language discrimination during pretraining reduces the multilingual gap on phonetic and higher-level linguistic measures while preserving cross-language sharing.

2. 2

   The strongest gains occur when language discrimination is introduced in the first HuBERT iteration; later or repeated interventions yield smaller improvements and increase language-wise segregation.

3. 3

   With language discrimination interventions, phone-ABX error decreased from 11.6% to 10.4% (monolingual: 10.8%), sWUGGY increased from 52.1% to 56.7% (monolingual: 58.5%), and ProsAudit lexical subtask increased from 68.9% to 72.9% (monolingual: 72.6%).

FAQ