Deep Learning over the Internet: Training Language Models Collaboratively

Summary

Pretraining state-of-the-art Transformer language models typically requires vast computing resources and expensive hardware clusters that are inaccessible to many researchers. Standard distributed deep learning struggles over public Internet infrastructure because consumer connections suffer from limited bandwidth, dynamic peer disconnections, and restrictive firewalls. To resolve these networking bottlenecks, Distributed Deep Learning in Open Collaborations (DeDLOC) introduces an adaptive training framework that accumulates massive batch sizes across heterogeneous volunteer devices before applying optimizer updates. The framework splits gradient vectors dynamically based on participant connection speeds and allows firewalled nodes to contribute data without performing aggregation. In a practical deployment with 40 volunteers, the 18-million-parameter sahajBERT model achieved Bengali downstream benchmark performance comparable to much larger multilingual models trained on dedicated supercomputers.

Context

Pretraining large Transformer language models demands extensive hardware resources that often exceed the budget of individual researchers and smaller organizations. While aggregating resources across consumer hardware or smaller clusters is conceptually possible, distributed deep learning over standard Internet connections struggles with slow bandwidth, network latency, dynamic disconnects, and firewall restrictions.

Approach / What changed

The Distributed Deep Learning in Open Collaborations (DeDLOC) algorithm accumulates a large batch across heterogeneous volunteer devices prior to each optimizer step, providing built-in fault tolerance against peer disconnects. It employs an adaptive gradient averaging mechanism that dynamically splits gradient vectors according to each peer's network bandwidth and accommodates firewalled nodes.

Takeaways

• DeDLOC accumulates gradients for a single large batch across all volunteer devices before each optimizer step, allowing training to proceed and adjust smoothly when peers join or disconnect.
• The adaptive averaging algorithm splits gradient vectors dynamically according to peer network speeds, assigning larger aggregation chunks to faster connections while letting firewalled nodes submit gradients without aggregating.
• Pretrained with 40 volunteers using DeDLOC, the 18M-parameter sahajBERT scored 95.45 F1 on Bengali NER and 91.97 accuracy on NCC, matching the performance of the 559M-parameter XLM-R Large.

Related reading

• Hugging Face · Dec 18, 2025

  Tokenization in Transformers v5: Simpler, Clearer, and More Modular

  Transformers v5 overhauls its tokenization framework by separating tokenizer architecture from trained vocabularies. In contrast to v4's dual slow Python and fast Rust files, v5 consolidates each model tokenizer into a single file defaulting to the Rust-backed TokenizersBackend. The pipeline stages—normalizer, pre-tokenizer, model algorithm such as BPE or Unigram, post-processor, and decoder—are now directly exposed and configurable rather than buried in serialized files. Practitioners can instantiate blank tokenizer architectures and train custom vocabularies directly from iterators using native methods like train_new_from_iterator while retaining model-specific formatting rules. The wrapper layer continues to bridge raw tokenization and model requirements by managing chat templates, context limits, and special token insertion.

  Ita Zaporozhets, Aritra Roy Gosthipaty, Arthur Zucker, Sergio Paniego, merve, Pedro Cuenca

• NVIDIA Developer Blog · Jul 6, 2026