Multilingual Knowledge Transfer under Data Constraints via Lexical Interventions

Summary

Cross-lingual knowledge transfer is critical for training multilingual language models when target-language data is scarce and downstream tasks demand reasoning or world knowledge. Existing transfer approaches often depend on scarce resources such as large parallel corpora, translation systems, auxiliary models, or extra training phases. To address this limitation, the authors introduce LINK, a data-level intervention method that operates directly during pretraining via bilingual vocabularies. LINK swaps randomly selected words in a portion of the high-resource English corpus with word-level translations according to a specified replacement ratio. Across evaluations spanning eight languages and five model sizes, the technique delivered notable performance gains on downstream target-language tasks while achieving up to a 2x speedup in training to reach equivalent performance.

Context

Building high-performing multilingual language models for languages with scarce training data requires transferring knowledge from high-resource languages for tasks like scientific reasoning, commonsense inference, and world knowledge. Existing methods for cross-lingual knowledge transfer depend on large parallel corpora, full translation systems, auxiliary models, or additional training stages that are unavailable for many low-resource languages.

Approach / What changed

The authors propose LINK, a data-level intervention method applied during model pretraining. LINK uses low-cost bilingual vocabularies to perform lexical substitutions within the high-resource English pretraining corpus. Based on a specified replacement ratio, randomly selected English words in a portion of the corpus are swapped with their word-level translations without requiring auxiliary models or extra training stages.

Takeaways

• Existing cross-lingual transfer approaches rely heavily on translation systems, parallel data, or auxiliary models that are largely unavailable for data-scarce target languages.
• LINK performs random word-level substitutions in the high-resource training corpus using bilingual vocabularies that can be acquired at near-zero cost for almost any language.
• In evaluations across eight languages and five model sizes, LINK delivered notable improvements on target-language downstream tasks and achieved up to a 2x training speedup to reach equivalent performance.

Related reading

• Apple · Aug 20, 2026

  Progressive Refinement: An Iterative Pseudo-Labeling Approach for Mandarin-English Code-Switching ASR

  Code-switching automatic speech recognition faces major performance challenges due to the scarcity of code-switching training data. To mitigate this data limitation, an iterative pseudo-labeling framework leverages a large unlabeled audio corpus to generate semi-supervised speech datasets. The system structure executes across three sequential phases: pseudo-label generation, two-stage bilingual model pre-training with supervised code-switching fine-tuning, and progressive iterative improvements. Successive refinement cycles further enhance the neural model's recognition accuracy when processing complex code-switching scenarios across alternating languages. When evaluated on the SEAME benchmark, this training pipeline achieved Mix Error Rate reductions of 6.35 percent on the devman subset and 8.29 percent on the devsge subset.

• Apple · Aug 20, 2026

  Scaling Laws for Mixture Pretraining Under Data Constraints

  Scaling language models demands large volumes of data, but high-value target sources like low-resource languages and specialized domains remain inherently scarce. Practitioners frequently combine limited target data with abundant generic data, creating a balance between domain underexposure and excessive repetition leading to overfitting. An empirical evaluation across more than 2,000 language-model training runs demonstrates that mixture training tolerates substantially higher data repetition than single-source training, allowing scarce target corpora to be repeated 15 to 20 times. Researchers introduced a repetition-aware mixture scaling law that accounts for both the diminishing returns of repeated target tokens and the regularizing effect of generic data. Optimizing this scaling law offers a principled framework for calculating effective mixture configurations and practical pretraining recommendations under severe data constraints.