Multi-vector embedding models store many token vectors per document, causing large indexes. Token pooling via Ward hierarchical clustering merges similar token embeddings into groups, reducing the number of vectors, but aggressive compression can lose retrieval-relevant distinctions. Pooling-aware training addresses this by averaging a retrieval loss on unpooled and pooled embeddings, using a straight-through estimator to pass gradients through the discrete clustering. At pool_factor=4, a normally trained model retains 94.6% of its unpooled nDCG@10, while pooling-aware training raises retention to 97.8% at the same 4x reduction in document vectors. Adaptive pooling with a calibrated threshold also benefits, retaining 97.3% of the unpooled score while storing 84.7% fewer vectors.
Context
Multi-vector embedding models store many token vectors per document, leading to large indexes. Token pooling via Ward hierarchical clustering merges similar token vectors to reduce storage, but aggressive compression can lose retrieval-relevant distinctions. Pooling-aware training addresses this by teaching the encoder to preserve useful information after merging.
Approach / What changed
Training the encoder with a loss that averages a retrieval loss on unpooled document embeddings with a loss on pooled embeddings, using a straight-through estimator to pass gradients through the discrete Ward clustering decisions. Implemented via MultiVectorTokenPoolingLoss in Sentence Transformers.
Takeaways
1
Token pooling via Ward hierarchical clustering merges similar token embeddings into groups represented by their mean, with controls like pool_factor, max_tokens, or distance_threshold to manage compression.
2
Pooling-aware training improves nDCG@10 retention from 94.6% to 97.8% at pool_factor=4, while storing 74.8% fewer document vectors than unpooled retrieval.
3
Adaptive pooling with a calibrated distance threshold can retain 97.3% of the pooling-aware model’s unpooled score while storing 84.7% fewer document vectors.
FAQ
How does token pooling work?
Token pooling uses Ward hierarchical clustering to merge similar token embeddings into groups, representing each group by its mean. The process builds a hierarchy of merges, and a pool_factor or distance threshold controls the level of compression.
What is pooling-aware training?
Pooling-aware training applies a retrieval objective to both unpooled and pooled document representations, averaging the losses. A straight-through estimator passes gradients from pooled vectors back to member tokens, teaching the encoder to preserve useful information after compression.
How do you calibrate adaptive pooling?
Adaptive pooling uses a shared distance threshold. You calibrate it by encoding a representative sample of documents, then using HierarchicalTokenPooling.calibrate_distance_threshold with a target average vector count to obtain a threshold for indexing.
Atlas Search finds matching document IDs fast, but materializing each hit from the collection is where the latency and CPU go. This trade-off map shows the benefit, the cost, and the condition for accepting that second phase.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.