---
title: "Tokenizers"
description: "1 posts about Tokenizers, summarised, each linking to the original."
---

# Tokenizers
> 1 posts about Tokenizers, summarised, each linking to the original.

## Articles

### [tokenizers v1: encode, decode and scaling, measured](https://yomu.fyi/post/tokenizers-v1-encode-decode-and-scaling-measured.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Arthur Zucker, Simon Brandeis, Luc Georges, Lysandre
- Published: Sep 21, 2026
- Type: Announcement

The tokenizers library is being reworked for version 1 with a heavy focus on performance, as tokenization can become a bottleneck that starves models of data when workloads scale. The work draws on ideas from other open-source tokenizers like gigatoken, tiktoken, and kitoken. Key optimizations include using SIMD instructions for UTF-8 text, ensuring merges never cross pre-token boundaries, and reusing scratch buffers to avoid allocations. The release candidate is implemented in the Rust pre-release on crates.io, with the crate split into tk-encode, tk-serialize, tk-convert, and tk-train. Batched model calls and faster decoding with parallel batch processing are also part of the improvements.