---
title: "Cross-store product clustering across millions of stores (2026) - Shopify"
description: "Shopify’s cross-store product clustering system groups listings of the same product across different merchants into a single global Universal Product Identifier. The approach avoids comparing every pair by generating candidate pairs using approximate nearest-neighbor search across billions of vector embeddings. Scoring uses three models of increasing cost: L1 is a fast GBDT classifier, L2 is a fine-tuned multilingual cross-encoder that reads both listings together, and L3 adds images. The UPI builder evaluates proposed merges using a trained model that weighs evidence, handling non-transitive pairwise judgments. The system caches scores and uses priority queues for expensive inference, enabling daily cluster updates while background GPU inference continues refining matches."
---

# Cross-store product clustering across millions of stores (2026) - Shopify

[Shopify](https://yomu.fyi/company/shopify) · Ilia Kuchumov · Oct 20, 2026

**Type:** Problem & solution

## Summary

Shopify’s cross-store product clustering system groups listings of the same product across different merchants into a single global Universal Product Identifier. The approach avoids comparing every pair by generating candidate pairs using approximate nearest-neighbor search across billions of vector embeddings. Scoring uses three models of increasing cost: L1 is a fast GBDT classifier, L2 is a fine-tuned multilingual cross-encoder that reads both listings together, and L3 adds images. The UPI builder evaluates proposed merges using a trained model that weighs evidence, handling non-transitive pairwise judgments. The system caches scores and uses priority queues for expensive inference, enabling daily cluster updates while background GPU inference continues refining matches.

## Context

Shopify hosts billions of product listings across millions of stores. Merchants structure the same real-world item differently (e.g., one product with all variants vs. separate products per color). The Shopify Catalog uses product clustering to group related listings under a single Universal Product Identifier (UPI). Intra-store clustering groups listings within a single store; cross-store clustering must group listings of the same product across different merchants.

## Approach / What changed

A three-stage pipeline: candidate generation via approximate nearest-neighbor search; scoring with three models (L1 GBDT for cheap, feature-based scoring; L2 fine-tuned multilingual E5 cross-encoder for joint text reading; L3 adds images and richer metadata); and a UPI builder that evaluates proposed merges using a trained model. Scored edges form a graph; clusters are built per component by a model weighing evidence for each proposed merge. LLM-generated labels train the specialized models, and cached scores plus priority queues keep the system up to date.

## Takeaways

- Cross-store clustering requires handling different merchant conventions: same product may appear with different titles, descriptions, pricing, and images across stores.
- Approximate nearest-neighbor search on product embeddings narrows the candidate space, but missed matches are acceptable because the listing stays searchable under its own identifier until a later build finds the connection.
- The system reuses expensive LLM calls: LLM-generated labels train cheaper specialized models, and completed scores are cached across daily cluster builds, focusing expensive inference only on new, changed, or previously unscored comparisons.

## FAQ

### How does joint encoding in L2 help compared to independent embeddings?

L2 is a fine-tuned multilingual E5 cross-encoder that reads both listings' titles, vendor names, predicted categories, URLs, and shop names together. Joint encoding lets it attend to differences between the two inputs, such as distinguishing 'for iPhone 15' from 'for iPhone 15 Pro,' which independent embeddings would miss.

### Why can't you use simple connected components for clustering?

Pairwise judgments can be non-transitive: the model might match A with B and B with C but reject A with C. Simple approaches like connected components would incorrectly group all three. The UPI builder uses a trained model to weigh evidence for and against each proposed merge, handling such intransitivity.

### How does the system keep up with a changing catalog?

New and changed products produce candidate pairs that are scored by L1, which is fast enough for the full set. L2 and L3 require GPU inference and are managed via priority queues. The UPI builder combines current L1 results with cached L2 and L3 scores, running daily while background inference continues processing new work.

**Tags:** [Catalog](https://yomu.fyi/topic/catalog), [Clustering](https://yomu.fyi/topic/clustering), [LLMs](https://yomu.fyi/topic/llm), [Machine Learning](https://yomu.fyi/topic/machine-learning), [Shopify](https://yomu.fyi/topic/shopify)

- Source: [Shopify](https://shopify.engineering/catalog-clustering-part-ii)
- Source URL: https://shopify.engineering/catalog-clustering-part-ii
- Ingested by Yomu: 2026-10-23T06:00:18.276Z

[Read original post](https://shopify.engineering/catalog-clustering-part-ii)
