Scaling Laws for Mixture Pretraining Under Data Constraints

Summary

Scaling language models demands large volumes of data, but high-value target sources like low-resource languages and specialized domains remain inherently scarce. Practitioners frequently combine limited target data with abundant generic data, creating a balance between domain underexposure and excessive repetition leading to overfitting. An empirical evaluation across more than 2,000 language-model training runs demonstrates that mixture training tolerates substantially higher data repetition than single-source training, allowing scarce target corpora to be repeated 15 to 20 times. Researchers introduced a repetition-aware mixture scaling law that accounts for both the diminishing returns of repeated target tokens and the regularizing effect of generic data. Optimizing this scaling law offers a principled framework for calculating effective mixture configurations and practical pretraining recommendations under severe data constraints.

Context

Language model scaling requires substantial data, but target datasets for specialized domains and low-resource languages are constrained in size. Mixing scarce target data with generic data poses a trade-off: insufficient target data causes underexposure, whereas excessive target data causes repetitive overfitting.

Approach / What changed

The authors analyzed over 2,000 language model training runs across multiple model sizes, target dataset sizes, and mixture types (multilingual, domain-specific, and quality-filtered) to develop a repetition-aware mixture scaling law that accounts for repeated target tokens and generic data regularization.

Takeaways

• Mixture training tolerates substantially higher repetition than single-source training, allowing scarce target corpora to be reused 15 to 20 times.
• The optimal number of target data repetitions depends on target data size, compute budget, and model scale.
• A repetition-aware mixture scaling law accounts for the decreasing value of repeated target tokens and the regularizing role of generic data to guide mixture configurations.

Related reading

• Two Sigma · Mar 21, 2024

  NeurIPS 2023: Our Favorite Papers on LLMs, Statistical Learning, and More

  Researchers reviewed prominent machine learning papers presented at NeurIPS 2023 covering large language models and statistical learning theory. One investigation showed that claimed emergent abilities in models such as GPT-3 often result from nonlinear evaluation metrics rather than fundamental shifts in model capability. To reduce the computational burden of model adaptation, QLoRA enables 65-billion-parameter model fine-tuning on a single 48-gigabyte GPU via 4-bit NormalFloat quantization and paged optimization. Direct Preference Optimization eliminates complex reward modeling by casting reinforcement learning from human feedback into a preference classification task. Additional work resolved statistical anomalies like double descent using effective parameter counts and introduced stochastic gradient approximations for Gaussian processes.

  Emily Majewski

• Apple · Aug 20, 2026