---
title: "tokenizers v1: encode, decode and scaling, measured"
description: "The tokenizers library is being reworked for version 1 with a heavy focus on performance, as tokenization can become a bottleneck that starves models of data when workloads scale. The work draws on ideas from other open-source tokenizers like gigatoken, tiktoken, and kitoken. Key optimizations include using SIMD instructions for UTF-8 text, ensuring merges never cross pre-token boundaries, and reusing scratch buffers to avoid allocations. The release candidate is implemented in the Rust pre-release on crates.io, with the crate split into tk-encode, tk-serialize, tk-convert, and tk-train. Batched model calls and faster decoding with parallel batch processing are also part of the improvements."
---

# tokenizers v1: encode, decode and scaling, measured

[Hugging Face](https://yomu.fyi/company/hugging-face) · Arthur Zucker, Simon Brandeis, Luc Georges, Lysandre · Sep 21, 2026

**Type:** Announcement

## Summary

The tokenizers library is being reworked for version 1 with a heavy focus on performance, as tokenization can become a bottleneck that starves models of data when workloads scale. The work draws on ideas from other open-source tokenizers like gigatoken, tiktoken, and kitoken. Key optimizations include using SIMD instructions for UTF-8 text, ensuring merges never cross pre-token boundaries, and reusing scratch buffers to avoid allocations. The release candidate is implemented in the Rust pre-release on crates.io, with the crate split into tk-encode, tk-serialize, tk-convert, and tk-train. Batched model calls and faster decoding with parallel batch processing are also part of the improvements.

## Context

The tokenizer has not historically been the bottleneck within ML workflows, but as models become faster and workloads scale, tokenization can starve the model of data. This work is in the Rust pre-release on crates.io.

## Approach / What changed

The approach involves using SIMD instructions for UTF-8 text, ensuring merges happen inside pre-tokens and never across boundaries, reusing scratch buffers for model memory, exposing post-processing as a pipeline stage, batching model calls, and decoding batches in parallel. The crate is split into tk-encode, tk-serialize, tk-convert, and tk-train.

## Takeaways

- Tokenization can become a bottleneck that starves models of data when training on massive datasets or serving many concurrent requests.
- SIMD instructions apply one operation to many bytes at once, suiting UTF-8 text well for faster tokenization.
- Merges happen inside a pre-token and never across the boundary between two of them, so the split decides what the rest of the pipeline sees.

## FAQ

### How does the tokenizer avoid CPU bottlenecks?

The tokenizer uses SIMD instructions to process many bytes at once, reuses scratch buffers to avoid allocations, and batches model calls to keep GPUs fed.

### What is the crate split for version 1?

The single crate is divided into tk-encode, tk-serialize, tk-convert, and tk-train, so an application links only what it needs.

### How is decoding made faster?

Decoded bytes are written directly into a reusable buffer, avoiding intermediate strings and copies, and batches are decoded in parallel.

**Tags:** [Performance](https://yomu.fyi/topic/performance), [Rust](https://yomu.fyi/topic/rust), [SIMD](https://yomu.fyi/topic/simd), [Tokenizers](https://yomu.fyi/topic/tokenizers)

- Source: [Hugging Face](https://huggingface.co/blog/tokenizers-v1)
- Source URL: https://huggingface.co/blog/tokenizers-v1
- Ingested by Yomu: 2026-09-21T15:00:09.155Z

[Read original post](https://huggingface.co/blog/tokenizers-v1)