# Porting fairseq wmt19 translation system to transformers

huggingface.co · Stas Bekman · Nov 3, 2020

**Type:** Tutorial

## Summary

The fairseq WMT19 news translation system was ported to the Hugging Face transformers library under the FSMT architecture. Development began with an initial proxy notebook emulating the transformers API before implementing native configuration, modeling, tokenization, and checkpoint conversion scripts. The author first targeted English-Russian models to handle complex dual-vocabulary requirements and directly evaluate output quality during testing, later extending the implementation to merged-vocabulary language pairs. Although model ensembling was omitted due to library limitations, removing fairseq optimizer states reduced the final model download footprint from 13GB to 1.1GB. The implementation was validated locally against converted fairseq checkpoints and merged into the main codebase after two weeks of iterative review.

## Context

Facebook FAIR's WMT19 translation system was hosted in the fairseq repository, requiring external dependencies like mosesdecoder and fastBPE alongside fairseq to run. Additionally, the original model checkpoints contained substantial unnecessary optimizer state data, creating a need to cleanly port and optimize the high-quality translators for the transformers ecosystem.

## Approach / What changed

The porting process began with a proxy notebook emulating the transformers API before building dedicated configuration, modeling, tokenization, and checkpoint conversion scripts based on existing BART modules. Initial porting targeted English-Russian models to establish support for dual vocabularies before accommodating merged-vocabulary pairs like German-English. Checkpoints were converted by stripping optimizer states and tested locally prior to upload and integration.

## Takeaways

- Original fairseq checkpoints included approximately 9GB of optimizer states across four checkpoints, allowing the converted transformers models to drop from 13GB to 1.1GB.
- Porting English-Russian models with two separate vocabularies of different sizes first made subsequently supporting merged-vocabulary models like German-English trivial.
- Fairseq model ensembles were not ported because the transformers library did not support ensembling.

**Tags:** [Machine Learning](https://yomu.fyi/topic/machine-learning), [Migrations](https://yomu.fyi/topic/migration), [Open Source](https://yomu.fyi/topic/open-source), [Python](https://yomu.fyi/topic/python)

[Read original post](https://huggingface.co/blog/porting-fsmt)