Topic

Tokenizers

1 post about Tokenizers. Every summary links to the original.

Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.

Newest first

Posts

Hugging Face
tokenizers v1: encode, decode and scaling, measured
The tokenizers library is being reworked for version 1 with a heavy focus on performance, as tokenization can become a bottleneck that starves models of data when workloads scale. The work draws on ideas from other open-source tokenizers like gigatoken, tiktoken, and kitoken. Key optimizations include using SIMD instructions for UTF-8 text, ensuring merges never cross pre-token boundaries, and reusing scratch buffers to avoid allocations. The release candidate is implemented in the Rust pre-release on crates.io, with the crate split into tk-encode, tk-serialize, tk-convert, and tk-train. Batched model calls and faster decoding with parallel batch processing are also part of the improvements.

The best engineering reads, once a week.

Topic

1 post about Tokenizers. Every summary links to the original.

Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.

Newest first

Hugging Face
tokenizers v1: encode, decode and scaling, measured
The tokenizers library is being reworked for version 1 with a heavy focus on performance, as tokenization can become a bottleneck that starves models of data when workloads scale. The work draws on ideas from other open-source tokenizers like gigatoken, tiktoken, and kitoken. Key optimizations include using SIMD instructions for UTF-8 text, ensuring merges never cross pre-token boundaries, and reusing scratch buffers to avoid allocations. The release candidate is implemented in the Rust pre-release on crates.io, with the crate split into tk-encode, tk-serialize, tk-convert, and tk-train. Batched model calls and faster decoding with parallel batch processing are also part of the improvements.

The best engineering reads, once a week.