# How to generate text: using different decoding methods for language generation with Transformers

huggingface.co · Patrick von Platen · Mar 1, 2020

**Type:** Tutorial

## Summary

Auto-regressive language generation relies heavily on decoding methods to convert predicted next-token probability distributions into coherent text. Greedy search selects the single most probable token at each timestep, but it frequently leads to repetitive phrasing and misses high-probability sequences hidden behind low-probability intermediate words. Beam search alleviates this issue by maintaining multiple parallel hypotheses and can be paired with n-gram penalties to reduce immediate repetitions. Sampling approaches, including top-K and top-p sampling, offer alternative paths that can yield more fluent, varied open-ended generation. Using the Hugging Face transformers library, practitioners can configure these strategies via parameters on the model generate function.

## Context

Open-ended auto-regressive language generation with transformer models often suffers from repetitive outputs and suboptimal token sequences depending on how next-word distributions are decoded.

## Approach / What changed

The Hugging Face transformers generate method is demonstrated using GPT-2 to compare greedy search, beam search with n-gram repetition penalties, and combined top-K and top-p sampling.

## Takeaways

- Greedy search picks the highest-probability token at each step but misses globally optimal sequences hidden behind lower-probability intermediate tokens.
- Beam search tracks multiple hypotheses across timesteps, and setting no\_repeat\_ngram\_size prevents identical n-grams from appearing more than once.
- Top-p and top-K sampling can be combined simultaneously in generate to filter out low-ranked tokens while preserving dynamic token candidate selection.

**Tags:** [LLMs](https://yomu.fyi/topic/llm), [Machine Learning](https://yomu.fyi/topic/machine-learning), [Open Source](https://yomu.fyi/topic/open-source), [Python](https://yomu.fyi/topic/python)

[Read original post](https://huggingface.co/blog/how-to-generate)