Leveraging Pre-trained Language Model Checkpoints for Encoder-Decoder Models

Summary

Pre-training sequence-to-sequence transformer models incurs massive computational costs, limiting development primarily to large institutions. To mitigate these expenses, encoder-decoder architectures can be warm-started using existing pre-trained checkpoints from encoder-only or decoder-only models like BERT and GPT-2. This walkthrough details the methodology and implementation of warm-starting sequence-to-sequence architectures using Hugging Face Transformers. By utilizing the EncoderDecoderModel framework alongside Seq2SeqTrainer, practitioners can construct and fine-tune models such as BERT2BERT on datasets like CNN/DailyMail. The resulting fully trained BERT2BERT model achieves a ROUGE-2 score of 18.22 on the full evaluation set, matching competitive sequence generation baselines at a fraction of standard pre-training costs.

Takeaways

1. 1

   Warm-starting encoder-decoder models from pre-trained checkpoints achieves competitive results to models like T5 and Pegasus at a fraction of the computational training cost.

2. 2

   Encoder-only architectures like BERT map inputs to fixed output dimensions, making standalone instances impractical for sequence-to-sequence tasks of unknown target lengths.

3. 3

   A fully fine-tuned BERT2BERT model evaluated on the CNN/DailyMail dataset achieves a ROUGE-2 score of 18.22.