Transformer-based Encoder-Decoder Models

Summary

Natural language generation tasks map input sequences to target sequences whose lengths cannot be known in advance and vary by content. Standard deep neural networks struggle with these variable mappings because their inputs and targets require fixed-dimensional vector representations. While recurrent neural networks addressed this challenge by generating target sequences auto-regressively from a compressed context state, transformer-based encoder-decoder architectures became the standard paradigm in modern natural language processing. The framework processes sequence-to-sequence problems by encoding source sequences and auto-regressively decoding target sequences token by token using conditional probability distributions. A step-by-step implementation demonstrates greedy decoding with Hugging Face Transformers, passing cached encoder hidden states alongside previously generated target tokens to iteratively generate German translations from English inputs.

Takeaways

1. 1

   Deep neural networks require fixed-dimension vector representations for inputs and targets, making recurrent or transformer-based encoder-decoder architectures necessary for variable-length sequence generation.

2. 2

   An auto-regressive decoder models the joint target sequence probability by decomposing it via Bayes' rule into a product of conditional probabilities for each successive target vector.

3. 3

   Greedy inference in Hugging Face Transformers iteratively generates target tokens by reusing cached encoder output states and appending argmax-sampled token IDs to the decoder input.