Leveraging Pre-trained Language Model Checkpoints for Encoder-Decoder Models

Summary

Pre-training sequence-to-sequence transformer models incurs massive computational costs, limiting development primarily to large institutions. To mitigate these expenses, encoder-decoder architectures can be warm-started using existing pre-trained checkpoints from encoder-only or decoder-only models like BERT and GPT-2. This walkthrough details the methodology and implementation of warm-starting sequence-to-sequence architectures using Hugging Face Transformers. By utilizing the EncoderDecoderModel framework alongside Seq2SeqTrainer, practitioners can construct and fine-tune models such as BERT2BERT on datasets like CNN/DailyMail. The resulting fully trained BERT2BERT model achieves a ROUGE-2 score of 18.22 on the full evaluation set, matching competitive sequence generation baselines at a fraction of standard pre-training costs.

Context

Pre-training massive encoder-decoder models like T5 and Pegasus requires enormous computational budgets, which restricts development to large companies and institutions.

Approach / What changed

Initialize sequence-to-sequence architectures by warm-starting the encoder and decoder components with existing pre-trained checkpoints (such as BERT or GPT-2) using Hugging Face's EncoderDecoderModel framework, then fine-tune them on target tasks with Seq2SeqTrainer.

Takeaways

• Warm-starting encoder-decoder models from pre-trained checkpoints achieves competitive results to models like T5 and Pegasus at a fraction of the computational training cost.
• Encoder-only architectures like BERT map inputs to fixed output dimensions, making standalone instances impractical for sequence-to-sequence tasks of unknown target lengths.
• A fully fine-tuned BERT2BERT model evaluated on the CNN/DailyMail dataset achieves a ROUGE-2 score of 18.22.

Related reading

• huggingface.co · Oct 10, 2020

  Transformer-based Encoder-Decoder Models

  Natural language generation tasks map input sequences to target sequences whose lengths cannot be known in advance and vary by content. Standard deep neural networks struggle with these variable mappings because their inputs and targets require fixed-dimensional vector representations. While recurrent neural networks addressed this challenge by generating target sequences auto-regressively from a compressed context state, transformer-based encoder-decoder architectures became the standard paradigm in modern natural language processing. The framework processes sequence-to-sequence problems by encoding source sequences and auto-regressively decoding target sequences token by token using conditional probability distributions. A step-by-step implementation demonstrates greedy decoding with Hugging Face Transformers, passing cached encoder hidden states alongside previously generated target tokens to iteratively generate German translations from English inputs.

  Patrick von Platen

• Nextdoor · Sep 7, 2023

  From Pre-trained to Fine-tuned: Nextdoor’s Path to Effective Embedding Applications

  Nextdoor transitioned its ranking and recommendation pipelines from traditional continuous and discrete interaction features to transformer-based representation learning. The engineering team deployed pre-trained Sentence-BERT models to generate multilingual post and comment representations, which were aggregated daily by interaction type to form user embeddings. To improve search recall, the team fine-tuned sentence transformers on unlabeled query session logs using contrastive learning and integrated HNSWlib for approximate nearest neighbor retrieval. Subsequent iterations incorporated labeled feedback, BERTopic for coarse personalization, and experiments with CLIP image embeddings. Infrastructure scaling challenges were addressed by performing embedding transformations directly within FeatureStore and optimizing feature payload formats to minimize microservice network bandwidth.