# Tokenization in Transformers v5: Simpler, Clearer, and More Modular

[Hugging Face](https://yomu.fyi/company/hugging-face) · Ita Zaporozhets, Aritra Roy Gosthipaty, Arthur Zucker, Sergio Paniego, merve, Pedro Cuenca · Dec 18, 2025

**Type:** Explainer

## Summary

Transformers v5 overhauls its tokenization framework by separating tokenizer architecture from trained vocabularies. In contrast to v4's dual slow Python and fast Rust files, v5 consolidates each model tokenizer into a single file defaulting to the Rust-backed TokenizersBackend. The pipeline stages—normalizer, pre-tokenizer, model algorithm such as BPE or Unigram, post-processor, and decoder—are now directly exposed and configurable rather than buried in serialized files. Practitioners can instantiate blank tokenizer architectures and train custom vocabularies directly from iterators using native methods like train\_new\_from\_iterator while retaining model-specific formatting rules. The wrapper layer continues to bridge raw tokenization and model requirements by managing chat templates, context limits, and special token insertion.

## Context

In Transformers v4, tokenizer pipeline architectures were obscured within serialized files, and tokenizers were split across duplicate Python and Rust implementations with distinct naming conventions and parity test suites, making custom inspection and domain-specific training cumbersome.

## Approach / What changed

Transformers v5 unifies tokenizers into a single file per model with the Rust-backed TokenizersBackend as the default, exposes pipeline components directly as class properties, and enables instantiating blank architecture templates for training custom vocabularies directly on new datasets.

## Takeaways

- Transformers v5 eliminates the parallel slow Python and fast Rust file split, adopting a single file per model where TokenizersBackend serves as the default Rust wrapper.
- Tokenization pipeline stages—normalizer, pre-tokenizer, model, post-processor, and decoder—are directly exposed as configurable properties on the tokenizer class.
- Developers can instantiate blank tokenizer architectures like LlamaTokenizer and train new vocabularies from data iterators while preserving original model formatting and conventions.

**Tags:** [LLMs](https://yomu.fyi/topic/llm), [Machine Learning](https://yomu.fyi/topic/machine-learning), [Open Source](https://yomu.fyi/topic/open-source), [Python](https://yomu.fyi/topic/python), [Rust](https://yomu.fyi/topic/rust)

[Read original post](https://huggingface.co/blog/tokenizers)