# Fit More and Train Faster With ZeRO via DeepSpeed and FairScale

huggingface.co · Stas Bekman · Jan 19, 2021

**Type:** Benchmark

## Summary

Machine learning model sizes frequently outpace consumer GPU memory, preventing practitioners from loading or fine-tuning large architectures on standard hardware. To address these resource constraints, the Hugging Face Trainer introduced experimental support for Zero Redundancy Optimizer features from DeepSpeed and FairScale starting in transformers v4.2.0. In dual-GPU benchmarks with a t5-large model, FairScale and DeepSpeed integrations expanded allowable batch sizes and shortened training durations compared to baseline DistributedDataParallel runs. Furthermore, DeepSpeed permitted single-GPU fine-tuning of a t5-3b model on a 24GB RTX-3090 card at batch size 20, whereas the baseline execution crashed immediately with an out-of-memory error. These optimizations manage memory allocation internally to curb memory fragmentation while requiring changes only to training scripts rather than model code.

## Context

Recent machine learning models grow faster than GPU memory capacity, leaving many users unable to train or load large models on available hardware, while model distillation efforts remain too slow.

## Approach / What changed

Hugging Face Trainer integrated ZeRO implementations from DeepSpeed and FairScale via experimental command-line flags (--deepspeed and --sharded\_ddp) to shard states, offload memory, and reduce GPU memory fragmentation without altering model architectures.

## Takeaways

- Starting in transformers v4.2.0, Hugging Face Trainer provides experimental support for FairScale via --sharded\_ddp and DeepSpeed via --deepspeed.
- On 2x 24GB Titan RTX GPUs training t5-large, DeepSpeed without CPU offloading increased maximum batch size from 16 to 40 while cutting training time from 30.95s to 10.40s.
- DeepSpeed enabled training a t5-3b model on a single 24GB RTX-3090 GPU at batch size 20, whereas standard single-GPU execution failed with an out-of-memory error even at batch size 1.

**Tags:** [Machine Learning](https://yomu.fyi/topic/machine-learning), [Open Source](https://yomu.fyi/topic/open-source), [Performance](https://yomu.fyi/topic/performance), [Python](https://yomu.fyi/topic/python)

[Read original post](https://huggingface.co/blog/zero-deepspeed-fairscale)