Loading…
Porting fairseq wmt19 translation system to transformers
Stas Bekman
Summary
The fairseq WMT19 news translation system was ported to the Hugging Face transformers library under the FSMT architecture. Development began with an initial proxy notebook emulating the transformers API before implementing native configuration, modeling, tokenization, and checkpoint conversion scripts. The author first targeted English-Russian models to handle complex dual-vocabulary requirements and directly evaluate output quality during testing, later extending the implementation to merged-vocabulary language pairs. Although model ensembling was omitted due to library limitations, removing fairseq optimizer states reduced the final model download footprint from 13GB to 1.1GB. The implementation was validated locally against converted fairseq checkpoints and merged into the main codebase after two weeks of iterative review.
Context
Facebook FAIR's WMT19 translation system was hosted in the fairseq repository, requiring external dependencies like mosesdecoder and fastBPE alongside fairseq to run. Additionally, the original model checkpoints contained substantial unnecessary optimizer state data, creating a need to cleanly port and optimize the high-quality translators for the transformers ecosystem.
Approach / What changed
The porting process began with a proxy notebook emulating the transformers API before building dedicated configuration, modeling, tokenization, and checkpoint conversion scripts based on existing BART modules. Initial porting targeted English-Russian models to establish support for dual vocabularies before accommodating merged-vocabulary pairs like German-English. Checkpoints were converted by stripping optimizer states and tested locally prior to upload and integration.
Takeaways
- Original fairseq checkpoints included approximately 9GB of optimizer states across four checkpoints, allowing the converted transformers models to drop from 13GB to 1.1GB.
- Porting English-Russian models with two separate vocabularies of different sizes first made subsequently supporting merged-vocabulary models like German-English trivial.
- Fairseq model ensembles were not ported because the transformers library did not support ensembling.
Related reading
mmBERT: ModernBERT goes Multilingual
mmBERT is a massively multilingual encoder model trained on more than 3T tokens across over 1,800 languages to improve upon existing multilingual architectures like XLM-R. Built upon ModernBERT with a Gemma 2 tokenizer, mmBERT employs a three-phase training curriculum consisting of pre-training on 60 languages, mid-training on 110 languages, and a final decay phase covering 1,833 languages. The training pipeline integrates an inverse mask ratio schedule, dynamic language temperature annealing, and TIES merging across three decay variants. Benchmark evaluations demonstrate strong natural language understanding on English GLUE and multilingual XTREME, as well as competitive retrieval performance on MTEB v2 and CoIR. The release includes standard base and small models alongside open training data and checkpoints.
Marc Marone, Orion Weller, William Fleshman, Eugene Yang, Dawn Lawrie, Ben Van Durmehuggingface.co ·
Summer at Hugging Face
Hugging Face released several major updates across its ecosystem, expanding its public Hub repository to over 16,000 models. Platform additions include Spaces for deploying Gradio and Streamlit demo applications, automatic TensorBoard instances, and evaluation metric tracking integrated with Papers With Code leaderboards. The Transformers library gained JAX/Flax support across more than 5,000 models, improved TensorFlow implementations, and introduced the transformers.onnx export module for model conversion. In research, the BigScience project completed large-scale training of a 13-billion-parameter English decoder model on Jean Zay, while the DeDLOC method enabled training the sahajBERT Bengali model without HPC infrastructure. Accepted conference papers detailed the Datasets library, prompt data-point equivalence, prompt-based fine-tuning heuristics, and block pruning techniques that reduced BERT size by 74 percent.