How to generate text: using different decoding methods for language generation with Transformers

Summary

Auto-regressive language generation relies heavily on decoding methods to convert predicted next-token probability distributions into coherent text. Greedy search selects the single most probable token at each timestep, but it frequently leads to repetitive phrasing and misses high-probability sequences hidden behind low-probability intermediate words. Beam search alleviates this issue by maintaining multiple parallel hypotheses and can be paired with n-gram penalties to reduce immediate repetitions. Sampling approaches, including top-K and top-p sampling, offer alternative paths that can yield more fluent, varied open-ended generation. Using the Hugging Face transformers library, practitioners can configure these strategies via parameters on the model generate function.

Context

Open-ended auto-regressive language generation with transformer models often suffers from repetitive outputs and suboptimal token sequences depending on how next-word distributions are decoded.

Approach / What changed

The Hugging Face transformers generate method is demonstrated using GPT-2 to compare greedy search, beam search with n-gram repetition penalties, and combined top-K and top-p sampling.

Takeaways

• Greedy search picks the highest-probability token at each step but misses globally optimal sequences hidden behind lower-probability intermediate tokens.
• Beam search tracks multiple hypotheses across timesteps, and setting no_repeat_ngram_size prevents identical n-grams from appearing more than once.
• Top-p and top-K sampling can be combined simultaneously in generate to filter out low-ranked tokens while preserving dynamic token candidate selection.

Related reading

• huggingface.co · Oct 10, 2020

  Transformer-based Encoder-Decoder Models

  Natural language generation tasks map input sequences to target sequences whose lengths cannot be known in advance and vary by content. Standard deep neural networks struggle with these variable mappings because their inputs and targets require fixed-dimensional vector representations. While recurrent neural networks addressed this challenge by generating target sequences auto-regressively from a compressed context state, transformer-based encoder-decoder architectures became the standard paradigm in modern natural language processing. The framework processes sequence-to-sequence problems by encoding source sequences and auto-regressively decoding target sequences token by token using conditional probability distributions. A step-by-step implementation demonstrates greedy decoding with Hugging Face Transformers, passing cached encoder hidden states alongside previously generated target tokens to iteratively generate German translations from English inputs.

  Patrick von Platen

• huggingface.co · Nov 9, 2020

  Leveraging Pre-trained Language Model Checkpoints for Encoder-Decoder Models

  Pre-training sequence-to-sequence transformer models incurs massive computational costs, limiting development primarily to large institutions. To mitigate these expenses, encoder-decoder architectures can be warm-started using existing pre-trained checkpoints from encoder-only or decoder-only models like BERT and GPT-2. This walkthrough details the methodology and implementation of warm-starting sequence-to-sequence architectures using Hugging Face Transformers. By utilizing the EncoderDecoderModel framework alongside Seq2SeqTrainer, practitioners can construct and fine-tune models such as BERT2BERT on datasets like CNN/DailyMail. The resulting fully trained BERT2BERT model achieves a ROUGE-2 score of 18.22 on the full evaluation set, matching competitive sequence generation baselines at a fraction of standard pre-training costs.