# Introducing 🤗 Accelerate

huggingface.co · Sylvain Gugger · Apr 16, 2021

**Type:** Announcement

## Summary

🤗 Accelerate enables PyTorch developers to execute raw training loops across CPUs, multi-GPU setups, and TPUs with minimal boilerplate modifications. Standard distributed training typically demands manual device assignments, DistributedSampler configurations, and divergent code paths that break single-device portability. By wrapping models, optimizers, and dataloaders through a unified prepare call, Accelerate automatically manages hardware placement, mixed precision operations, and sampler batch slicing without requiring custom distributed samplers. Distributed evaluation is streamlined using a gather utility that consolidates prediction tensors across active worker processes. Furthermore, a dedicated command-line interface provides an interactive questionnaire to store runtime configuration defaults and orchestrate local or AWS SageMaker runs.

## Context

Customizing standard PyTorch training loops for distributed execution across multi-GPU nodes or TPUs and enabling mixed precision requires extensive boilerplate code, such as manual device placement and DistributedSampler logic, which breaks compatibility when reverting to single CPU or single GPU setups.

## Approach / What changed

Accelerate provides an Accelerator class with unified methods (such as prepare, backward, and gather) to automatically handle device placement, wrapper containers, and mixed-precision operations, accompanied by a CLI tool to configure and launch distributed jobs.

## Takeaways

- Accelerate eliminates the need for DistributedSampler by wrapping dataloaders to extract process-specific indices or skip batches on IterableDataset instances.
- The library synchronizes random number generators across processes for the sampler by default, preserving identical shuffling while allowing distinct data augmentations per process.
- The accelerate config and accelerate launch CLI tools allow developers to save environment settings via an interactive prompt and execute training across hardware targets, including AWS SageMaker.

**Tags:** [AWS](https://yomu.fyi/topic/aws), [Machine Learning](https://yomu.fyi/topic/machine-learning), [Open Source](https://yomu.fyi/topic/open-source), [Python](https://yomu.fyi/topic/python)

[Read original post](https://huggingface.co/blog/accelerate-library)