Scaling Laws for Mixture Pretraining Under Data Constraints

Summary

Scaling language models demands large volumes of data, but high-value target sources like low-resource languages and specialized domains remain inherently scarce. Practitioners frequently combine limited target data with abundant generic data, creating a balance between domain underexposure and excessive repetition leading to overfitting. An empirical evaluation across more than 2,000 language-model training runs demonstrates that mixture training tolerates substantially higher data repetition than single-source training, allowing scarce target corpora to be repeated 15 to 20 times. Researchers introduced a repetition-aware mixture scaling law that accounts for both the diminishing returns of repeated target tokens and the regularizing effect of generic data. Optimizing this scaling law offers a principled framework for calculating effective mixture configurations and practical pretraining recommendations under severe data constraints.

Takeaways

1. 1

   Mixture training tolerates substantially higher repetition than single-source training, allowing scarce target corpora to be reused 15 to 20 times.

2. 2

   The optimal number of target data repetitions depends on target data size, compute budget, and model scale.

3. 3

   A repetition-aware mixture scaling law accounts for the decreasing value of repeated target tokens and the regularizing role of generic data to guide mixture configurations.