Loading…
Transformer-based Encoder-Decoder Models
Patrick von Platen
Summary
Natural language generation tasks map input sequences to target sequences whose lengths cannot be known in advance and vary by content. Standard deep neural networks struggle with these variable mappings because their inputs and targets require fixed-dimensional vector representations. While recurrent neural networks addressed this challenge by generating target sequences auto-regressively from a compressed context state, transformer-based encoder-decoder architectures became the standard paradigm in modern natural language processing. The framework processes sequence-to-sequence problems by encoding source sequences and auto-regressively decoding target sequences token by token using conditional probability distributions. A step-by-step implementation demonstrates greedy decoding with Hugging Face Transformers, passing cached encoder hidden states alongside previously generated target tokens to iteratively generate German translations from English inputs.
Context
Natural language generation tasks require mapping input sequences to variable-length target sequences where output length depends on content rather than fixed dimensions, which fixed-dimensional deep neural networks cannot natively model.
Approach / What changed
The transformer-based encoder-decoder model encodes input vector sequences and auto-regressively predicts next-token probability distributions using the encoder output and prior target tokens, implemented via PyTorch and Hugging Face Transformers for greedy decoding.
Takeaways
- Deep neural networks require fixed-dimension vector representations for inputs and targets, making recurrent or transformer-based encoder-decoder architectures necessary for variable-length sequence generation.
- An auto-regressive decoder models the joint target sequence probability by decomposing it via Bayes' rule into a product of conditional probabilities for each successive target vector.
- Greedy inference in Hugging Face Transformers iteratively generates target tokens by reusing cached encoder output states and appending argmax-sampled token IDs to the decoder input.
Related reading
huggingface.co ·
The Reformer - Pushing the limits of language modeling
Standard transformer models hit memory bottlenecks on long sequence modeling tasks due to the quadratic asymptotic memory complexity of global self-attention and oversized positional embedding matrices. The Reformer architecture overcomes these constraints to train sequences of up to half a million tokens using under 8GB of RAM. It re-engineers transformer operations using local and Locality Sensitive Hashing self-attention, chunked feed forward layers, reversible residual layers, and axial positional encodings. In empirical benchmarks using google/reformer-crime-and-punishment, axial positional encodings reduce the model parameter count from over 136 million to approximately 2.58 million by factorizing the positional dimensions. This architectural change cuts inference memory consumption from 959 MB down to 447 MB for evaluated benchmark workloads.
Patrick von Platenhuggingface.co ·
Leveraging Pre-trained Language Model Checkpoints for Encoder-Decoder Models
Pre-training sequence-to-sequence transformer models incurs massive computational costs, limiting development primarily to large institutions. To mitigate these expenses, encoder-decoder architectures can be warm-started using existing pre-trained checkpoints from encoder-only or decoder-only models like BERT and GPT-2. This walkthrough details the methodology and implementation of warm-starting sequence-to-sequence architectures using Hugging Face Transformers. By utilizing the EncoderDecoderModel framework alongside Seq2SeqTrainer, practitioners can construct and fine-tune models such as BERT2BERT on datasets like CNN/DailyMail. The resulting fully trained BERT2BERT model achieves a ROUGE-2 score of 18.22 on the full evaluation set, matching competitive sequence generation baselines at a fraction of standard pre-training costs.