Mix, Don't Tune: Bilingual Pre-Training Outperforms Hyperparameter Search in Data-Constrained Settings

Summary

In data-constrained language model pre-training, two remedies exist: aggressive hyperparameter tuning such as high weight decay, and mixing in data from a high-resource auxiliary language to expand the training distribution. This study systematically compares these levers across four model scales from 150M to 1.43B parameters, using Arabic as the low-resource target and English as the auxiliary, over approximately 1000 pre-training runs. Mixing yields larger improvements than hyperparameter tuning on both validation loss and downstream task accuracy, and the gap grows with model size. Mixing boosts performance by an amount equivalent to 2–3× the unique target data on validation loss and 2–13× on downstream task accuracy, with the gain scaling steeply with model size. The authors recommend mixing in a high-resource language, prioritizing the mixing ratio over hyperparameter tuning, and transferring hyperparameters from a small proxy model via μP, noting that target-language validation loss systematically underestimates mixing’s value.

Takeaways

1. 1

   Mixing in data from a high-resource auxiliary language yields larger improvements than aggressive hyperparameter tuning on both validation loss and downstream task accuracy, with the gap widening as model size increases.

2. 2

   The performance boost from mixing is equivalent to 2–3× the unique target data on validation loss and 2–13× on downstream task accuracy, scaling steeply with model size.

3. 3

   Target-language validation loss systematically underestimates the value of mixing because it only captures regularization from diversified training signal, not the additional knowledge contributed by auxiliary data.

FAQ