# Leveraging Pre-trained Language Model Checkpoints for Encoder-Decoder Models

huggingface.co · Patrick von Platen · Nov 9, 2020

**Type:** Tutorial

## Summary

Pre-training sequence-to-sequence transformer models incurs massive computational costs, limiting development primarily to large institutions. To mitigate these expenses, encoder-decoder architectures can be warm-started using existing pre-trained checkpoints from encoder-only or decoder-only models like BERT and GPT-2. This walkthrough details the methodology and implementation of warm-starting sequence-to-sequence architectures using Hugging Face Transformers. By utilizing the EncoderDecoderModel framework alongside Seq2SeqTrainer, practitioners can construct and fine-tune models such as BERT2BERT on datasets like CNN/DailyMail. The resulting fully trained BERT2BERT model achieves a ROUGE-2 score of 18.22 on the full evaluation set, matching competitive sequence generation baselines at a fraction of standard pre-training costs.

## Context

Pre-training massive encoder-decoder models like T5 and Pegasus requires enormous computational budgets, which restricts development to large companies and institutions.

## Approach / What changed

Initialize sequence-to-sequence architectures by warm-starting the encoder and decoder components with existing pre-trained checkpoints (such as BERT or GPT-2) using Hugging Face's EncoderDecoderModel framework, then fine-tune them on target tasks with Seq2SeqTrainer.

## Takeaways

- Warm-starting encoder-decoder models from pre-trained checkpoints achieves competitive results to models like T5 and Pegasus at a fraction of the computational training cost.
- Encoder-only architectures like BERT map inputs to fixed output dimensions, making standalone instances impractical for sequence-to-sequence tasks of unknown target lengths.
- A fully fine-tuned BERT2BERT model evaluated on the CNN/DailyMail dataset achieves a ROUGE-2 score of 18.22.

**Tags:** [LLMs](https://yomu.fyi/topic/llm), [Machine Learning](https://yomu.fyi/topic/machine-learning), [Open Source](https://yomu.fyi/topic/open-source), [Python](https://yomu.fyi/topic/python)

[Read original post](https://huggingface.co/blog/warm-starting-encoder-decoder)