# Transformer-based Encoder-Decoder Models

huggingface.co · Patrick von Platen · Oct 10, 2020

**Type:** Explainer

## Summary

Natural language generation tasks map input sequences to target sequences whose lengths cannot be known in advance and vary by content. Standard deep neural networks struggle with these variable mappings because their inputs and targets require fixed-dimensional vector representations. While recurrent neural networks addressed this challenge by generating target sequences auto-regressively from a compressed context state, transformer-based encoder-decoder architectures became the standard paradigm in modern natural language processing. The framework processes sequence-to-sequence problems by encoding source sequences and auto-regressively decoding target sequences token by token using conditional probability distributions. A step-by-step implementation demonstrates greedy decoding with Hugging Face Transformers, passing cached encoder hidden states alongside previously generated target tokens to iteratively generate German translations from English inputs.

## Context

Natural language generation tasks require mapping input sequences to variable-length target sequences where output length depends on content rather than fixed dimensions, which fixed-dimensional deep neural networks cannot natively model.

## Approach / What changed

The transformer-based encoder-decoder model encodes input vector sequences and auto-regressively predicts next-token probability distributions using the encoder output and prior target tokens, implemented via PyTorch and Hugging Face Transformers for greedy decoding.

## Takeaways

- Deep neural networks require fixed-dimension vector representations for inputs and targets, making recurrent or transformer-based encoder-decoder architectures necessary for variable-length sequence generation.
- An auto-regressive decoder models the joint target sequence probability by decomposing it via Bayes' rule into a product of conditional probabilities for each successive target vector.
- Greedy inference in Hugging Face Transformers iteratively generates target tokens by reusing cached encoder output states and appending argmax-sampled token IDs to the decoder input.

**Tags:** [LLMs](https://yomu.fyi/topic/llm), [Machine Learning](https://yomu.fyi/topic/machine-learning), [Open Source](https://yomu.fyi/topic/open-source), [Python](https://yomu.fyi/topic/python)

[Read original post](https://huggingface.co/blog/encoder-decoder)