# Deep Learning over the Internet: Training Language Models Collaboratively

huggingface.co · Max Ryabinin, Lucile Saulnier · Jul 15, 2021

**Type:** Problem & solution

## Summary

Pretraining state-of-the-art Transformer language models typically requires vast computing resources and expensive hardware clusters that are inaccessible to many researchers. Standard distributed deep learning struggles over public Internet infrastructure because consumer connections suffer from limited bandwidth, dynamic peer disconnections, and restrictive firewalls. To resolve these networking bottlenecks, Distributed Deep Learning in Open Collaborations (DeDLOC) introduces an adaptive training framework that accumulates massive batch sizes across heterogeneous volunteer devices before applying optimizer updates. The framework splits gradient vectors dynamically based on participant connection speeds and allows firewalled nodes to contribute data without performing aggregation. In a practical deployment with 40 volunteers, the 18-million-parameter sahajBERT model achieved Bengali downstream benchmark performance comparable to much larger multilingual models trained on dedicated supercomputers.

## Context

Pretraining large Transformer language models demands extensive hardware resources that often exceed the budget of individual researchers and smaller organizations. While aggregating resources across consumer hardware or smaller clusters is conceptually possible, distributed deep learning over standard Internet connections struggles with slow bandwidth, network latency, dynamic disconnects, and firewall restrictions.

## Approach / What changed

The Distributed Deep Learning in Open Collaborations (DeDLOC) algorithm accumulates a large batch across heterogeneous volunteer devices prior to each optimizer step, providing built-in fault tolerance against peer disconnects. It employs an adaptive gradient averaging mechanism that dynamically splits gradient vectors according to each peer's network bandwidth and accommodates firewalled nodes.

## Takeaways

- DeDLOC accumulates gradients for a single large batch across all volunteer devices before each optimizer step, allowing training to proceed and adjust smoothly when peers join or disconnect.
- The adaptive averaging algorithm splits gradient vectors dynamically according to peer network speeds, assigning larger aggregation chunks to faster connections while letting firewalled nodes submit gradients without aggregating.
- Pretrained with 40 volunteers using DeDLOC, the 18M-parameter sahajBERT scored 95.45 F1 on Bengali NER and 91.97 accuracy on NCC, matching the performance of the 559M-parameter XLM-R Large.

**Tags:** [LLMs](https://yomu.fyi/topic/llm), [Machine Learning](https://yomu.fyi/topic/machine-learning), [Open Source](https://yomu.fyi/topic/open-source), [Scalability](https://yomu.fyi/topic/scalability)

[Read original post](https://huggingface.co/blog/collaborative-training)