---
title: "Hyperparameter Tuning"
description: "1 posts about Hyperparameter Tuning, summarised, each linking to the original."
---

# Hyperparameter Tuning
> 1 posts about Hyperparameter Tuning, summarised, each linking to the original.

## Articles

### [Mix, Don't Tune: Bilingual Pre-Training Outperforms Hyperparameter Search in Data-Constrained Settings](https://yomu.fyi/post/mix-don-t-tune-bilingual-pre-training-outperforms-hyperparameter-searc.md)
- Company: [Apple](https://yomu.fyi/company/apple.md)
- Published: Oct 13, 2026
- Type: Benchmark

In data-constrained language model pre-training, two remedies exist: aggressive hyperparameter tuning such as high weight decay, and mixing in data from a high-resource auxiliary language to expand the training distribution. This study systematically compares these levers across four model scales from 150M to 1.43B parameters, using Arabic as the low-resource target and English as the auxiliary, over approximately 1000 pre-training runs. Mixing yields larger improvements than hyperparameter tuning on both validation loss and downstream task accuracy, and the gap grows with model size. Mixing boosts performance by an amount equivalent to 2–3× the unique target data on validation loss and 2–13× on downstream task accuracy, with the gain scaling steeply with model size. The authors recommend mixing in a high-resource language, prioritizing the mixing ratio over hyperparameter tuning, and transferring hyperparameters from a small proxy model via μP, noting that target-language validation loss systematically underestimates mixing’s value.