IDEA Prune: An Integrated Enlarge-and-Prune Pipeline in Generative Language Model Pretraining

Summary

Recent advancements in large language models necessitate efficient deployment options within constrained inference budgets. While structured pruning offers better token efficiency than training target-sized models from scratch, previous work frequently ignores the pretraining of enlarged models. To resolve this gap, researchers developed an integrated enlarge-and-prune pipeline that unifies enlarged model training, pruning, and recovery under a single cosine annealing learning rate schedule. This system incorporates an iterative structured pruning technique to gradually eliminate parameters, redistributing model capacity across surviving neurons and avoiding the knowledge loss seen in naive approaches. Evaluated on compressing 2.8B parameter models to 1.3B across up to 2T pretraining tokens, the approach yields superior downstream model performance.

Context

Large language model deployment faces limited inference budgets, making token-efficient model compression critical. Although structured pruning pipelines offer efficiency benefits over training target-size models from scratch, prior works often overlook enlarged model pretraining, while naive enlarge-and-prune pipelines suffer knowledge loss from rising learning rates.

Approach / What changed

The authors propose an integrated enlarge-and-prune pipeline that combines enlarged model training, pruning, and recovery under a single cosine annealing learning rate schedule. This framework is paired with an iterative structured pruning method for gradual parameter removal, which redistributes model capacity among surviving neurons and mitigates knowledge loss.

Takeaways

• An integrated pipeline unifies enlarged model pretraining, pruning, and recovery under a single cosine annealing learning rate schedule.
• Iterative structured pruning enables gradual parameter removal to redistribute model capacity among surviving neurons and mitigate knowledge loss.
• Compressing 2.8B parameter models to 1.3B with up to 2T pretraining tokens demonstrates token efficiency and superior pruned model performance.

Related reading

• Amazon · May 15, 2026

  Making LLMs faster without sacrificing accuracy

  Standard scaling laws optimize parameter count and training data budgets to minimize loss but ignore internal Transformer architecture configurations. To address throughput disparities among equally sized models, researchers developed a scaling framework incorporating hidden size, the multilayer perceptron to attention parameter ratio, and grouped-query attention. The method calibrates a correction surface over standard Chinchilla scaling laws to independently optimize separable architectural parameters for target accuracy. Evaluated across more than 200 models up to three billion parameters, the framework produced the Panda model family for maximum accuracy and the Surefire family for Pareto efficiency. Surefire models matched or exceeded LLaMA-3.2 accuracy while increasing inference throughput by 12% to 47% across vLLM and SGLang serving systems on modern GPUs.

  Tao Yu, Youngsuk Park

• Pinterest · May 1, 2026

  Optimizing ML Workload Network Efficiency (Part I): Feature Trimmer