# Accelerate Large Model Training using DeepSpeed

huggingface.co · Sourab Mangrulkar, Sylvain Gugger · Jun 28, 2022

**Type:** Tutorial

## Summary

Training large models on hardware with limited GPU memory frequently causes out-of-memory errors when using standard Distributed Data Parallel. To solve this bottleneck, Hugging Face Accelerate integrates DeepSpeed ZeRO data parallelism to shard optimizer states, gradients, and model parameters across workers. For a 900-million-parameter DeBERTa model, ZeRO Stage 2 increased the maximum per-device batch size from eight to forty while achieving a 3.5-fold training speedup over DDP without degrading accuracy or F1 score. Advanced configurations allow sequence-to-sequence chatbot finetuning and full ZeRO Stage 3 CPU offloading for models like the 1.5-billion-parameter GPT-XL, enabling batch size 16 training where DDP fails. Accelerate enables these memory optimizations through simple configuration files and the accelerate launch command with minimal or zero code modifications.

## Context

Training large transformer models on standard GPU hardware frequently triggers out-of-memory errors due to optimizer states, gradients, and model parameters exceeding GPU memory capacity. Standard Distributed Data Parallel cannot handle large batch sizes on memory-constrained setups and fails entirely on 1.5-billion-parameter models like GPT-XL even with a batch size of one.

## Approach / What changed

Hugging Face Accelerate integrates DeepSpeed Zero Redundancy Optimizer features to shard optimizer states, gradients, and model parameters across GPUs or offload them to CPU memory. Users configure DeepSpeed ZeRO Stage 2 or Stage 3 via Accelerate plugins or JSON configuration files, then launch training scripts using the accelerate launch command without extensive code refactoring.

## Takeaways

- DeepSpeed ZeRO Stage 2 enabled a 5X larger batch size (40 vs. 8) and a ~3.5X training speedup compared to DDP on a 900M DeBERTa model across two 24GB GPUs.
- ZeRO Stage 3 CPU offloading allows training a 1.5B parameter GPT-XL model with a batch size of 16 on hardware where DDP failed with out-of-memory errors at batch size 1.
- Accelerate supports DeepSpeed ZeRO configurations through CLI prompts or custom JSON files, requiring minimal or no modifications to training scripts.

**Tags:** [LLMs](https://yomu.fyi/topic/llm), [Machine Learning](https://yomu.fyi/topic/machine-learning), [Open Source](https://yomu.fyi/topic/open-source), [Performance](https://yomu.fyi/topic/performance), [Python](https://yomu.fyi/topic/python)

[Read original post](https://huggingface.co/blog/accelerate-deepspeed)