MetaBy Carl Hu, Justin Johnson, Parish Aggarwal, Kaushik Rangadurai, Senthil Manickavelu, Tianwei ZhangExplainer
GenRecs Tokenization replaces per-item IDs with a stable semantic vocabulary of tokens that describe what an item is, enabling generative recommender models to reason about ad content and user preferences together. The tokenization pipeline encodes each item into a semantic embedding, decomposes it into independent subspaces, and quantizes each subspace to a codebook entry, producing discrete tokens that similar items share. This approach accelerates learning for new entities with no engagement history, enables cross-campaign transfer, and allows the system to benefit from LLM scaling and world knowledge. In production, paired with Meta’s Generative Ads Model and Adaptive Ranking Model, GenRecs Tokenization delivered a +0.35% lift in conversions across surfaces. The tokenization flywheel improves tail ad predictions, which enriches training data and further improves predictions for all ads, creating a self-reinforcing feedback loop.
Meta’s ads recommendation system traditionally used per-item IDs, which limited generalization for new entities with no engagement history. GenRecs Tokenization introduces a semantic token vocabulary to enable generative recommender models to reason about ad content and user preferences together.
The three-stage tokenization pipeline: 1) Encode each item into a semantic embedding space. 2) Decompose that embedding into independent subspaces. 3) Quantize by mapping each subspace to a codebook entry. For many ad representations, it uses prefix n-grams combining a learned encoder, residual decomposition, learned vector codebooks, and token parametrization.
GenRecs Tokenization represents each item with a set of tokens from a stable semantic vocabulary instead of a dedicated ID, allowing the model to relate new items to known categories.
The tokenization pipeline consists of encoding items into a semantic embedding, decomposing into independent subspaces, and quantizing each subspace to a codebook entry.
In production, GenRecs Tokenization delivered a +0.35% lift in conversions when paired with Meta’s GEM and Adaptive Ranking Model.
It uses semantic tokens that describe what the item is, so a new trail shoe inherits what the model learned about that category from previous season's products, enabling cold-start generalization.
Cluster purity, distribution balance, and codebook utilization. Items that share a code tend to share a category, items spread evenly across semantic IDs, and a larger share of codes holds at least one item.
By improving tail ad prediction quality, it enriches training data, which then improves prediction quality for all ads, creating a self-reinforcing feedback loop.
Summaries are generated from the original by a language model. Check the original before relying on it.
See the idea
A practical how-to for engineers who need to turn brand from a style guide into a governed signal pipeline: anchor on jobs, model brand roles, unify signals, tokenize meaning, and close the loop with measurement.
Keep reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
One useful email. Unsubscribe anytime.
