Leveraging multimodal LLMs for Shopify’s global catalogue: Recap of expo talk at ICLR 2025

Summary

Shopify’s Global Catalogue addresses fragmented, multilingual, multimodal product data that makes semantic search, faceting, matching, and AI-driven discovery difficult across millions of merchants. The system combines product data, understanding, matching, and reconciliation layers to standardize metadata, identify duplicate listings, and create canonical records with Universal Product IDs. Shopify fine-tunes vision large language models for multiple tasks per catalogue entity, while streaming infrastructure processes more than 10 million product updates daily and serves 40 million multimodal LLM-powered inferences. The resulting data supports merchant-admin suggestions, search, recommendations, embeddings, and conversational commerce, although scalability, accuracy, latency, model consolidation, graph reasoning, and continuous retraining remain active challenges.

Takeaways

1. 1

   Multi-task, multi-entity vision LLMs perform classification, attribute extraction, image understanding, title standardization, description analysis, and review summarization; the post reports that task interdependencies improve inference quality over siloed models.

2. 2

   Product matching combines locality-sensitive hashing, embeddings, probabilistic methods, deterministic signals, and discriminator-based edge pruning before connected components receive Universal Product IDs.

3. 3

   Selective field prompting reduced median latency from 2 seconds to 500 milliseconds and GPU token usage by 40 percent, while FP8 quantization enabled larger batch sizes without sacrificing inference accuracy.