# Scaling Laws for Mixture Pretraining Under Data Constraints

[Apple](https://yomu.fyi/company/apple) · Aug 20, 2026

**Type:** Problem & solution

## Summary

Scaling language models demands large volumes of data, but high-value target sources like low-resource languages and specialized domains remain inherently scarce. Practitioners frequently combine limited target data with abundant generic data, creating a balance between domain underexposure and excessive repetition leading to overfitting. An empirical evaluation across more than 2,000 language-model training runs demonstrates that mixture training tolerates substantially higher data repetition than single-source training, allowing scarce target corpora to be repeated 15 to 20 times. Researchers introduced a repetition-aware mixture scaling law that accounts for both the diminishing returns of repeated target tokens and the regularizing effect of generic data. Optimizing this scaling law offers a principled framework for calculating effective mixture configurations and practical pretraining recommendations under severe data constraints.

## Context

Language model scaling requires substantial data, but target datasets for specialized domains and low-resource languages are constrained in size. Mixing scarce target data with generic data poses a trade-off: insufficient target data causes underexposure, whereas excessive target data causes repetitive overfitting.

## Approach / What changed

The authors analyzed over 2,000 language model training runs across multiple model sizes, target dataset sizes, and mixture types (multilingual, domain-specific, and quality-filtered) to develop a repetition-aware mixture scaling law that accounts for repeated target tokens and generic data regularization.

## Takeaways

- Mixture training tolerates substantially higher repetition than single-source training, allowing scarce target corpora to be reused 15 to 20 times.
- The optimal number of target data repetitions depends on target data size, compute budget, and model scale.
- A repetition-aware mixture scaling law accounts for the decreasing value of repeated target tokens and the regularizing role of generic data to guide mixture configurations.

**Tags:** [LLMs](https://yomu.fyi/topic/llm), [Machine Learning](https://yomu.fyi/topic/machine-learning)

[Read original post](https://machinelearning.apple.com/research/scaling-laws-mixture-pretraining)