# IDEA Prune: An Integrated Enlarge-and-Prune Pipeline in Generative Language Model Pretraining

[Apple](https://yomu.fyi/company/apple) · Aug 26, 2026

**Type:** Problem & solution

## Summary

Recent advancements in large language models necessitate efficient deployment options within constrained inference budgets. While structured pruning offers better token efficiency than training target-sized models from scratch, previous work frequently ignores the pretraining of enlarged models. To resolve this gap, researchers developed an integrated enlarge-and-prune pipeline that unifies enlarged model training, pruning, and recovery under a single cosine annealing learning rate schedule. This system incorporates an iterative structured pruning technique to gradually eliminate parameters, redistributing model capacity across surviving neurons and avoiding the knowledge loss seen in naive approaches. Evaluated on compressing 2.8B parameter models to 1.3B across up to 2T pretraining tokens, the approach yields superior downstream model performance.

## Context

Large language model deployment faces limited inference budgets, making token-efficient model compression critical. Although structured pruning pipelines offer efficiency benefits over training target-size models from scratch, prior works often overlook enlarged model pretraining, while naive enlarge-and-prune pipelines suffer knowledge loss from rising learning rates.

## Approach / What changed

The authors propose an integrated enlarge-and-prune pipeline that combines enlarged model training, pruning, and recovery under a single cosine annealing learning rate schedule. This framework is paired with an iterative structured pruning method for gradual parameter removal, which redistributes model capacity among surviving neurons and mitigates knowledge loss.

## Takeaways

- An integrated pipeline unifies enlarged model pretraining, pruning, and recovery under a single cosine annealing learning rate schedule.
- Iterative structured pruning enables gradual parameter removal to redistribute model capacity among surviving neurons and mitigate knowledge loss.
- Compressing 2.8B parameter models to 1.3B with up to 2T pretraining tokens demonstrates token efficiency and superior pruned model performance.

**Tags:** [LLMs](https://yomu.fyi/topic/llm), [Machine Learning](https://yomu.fyi/topic/machine-learning), [Performance](https://yomu.fyi/topic/performance)

[Read original post](https://machinelearning.apple.com/research/idea-prune-pipeline)