Using Rich Image and Text Data to Categorize Products at Scale

Summary

Shopify revisited product categorization as its platform expanded to millions of merchants selling billions of products across diverse industries. The model uses product titles, descriptions, types, vendors, collections, tags, and images, converting them into embeddings with Multilingual BERT and MobileNet-V2 before hierarchical classification across the seven-level Google Product Taxonomy. Each taxonomy level is treated as a separate multi-class task, with parent predictions fed into child predictions and weighted losses combined during training; confidence thresholds filter predictions at each level. The upgraded system increased leaf precision by 8% while almost doubling coverage, although thresholds require balancing accuracy, coverage, merchant experience, and performance in sensitive categories, and future work includes addressing data imbalance and adding merchant-level features.

Context

Shopify needed to reevaluate product categorization as its merchant and product volume grew. The existing model used only textual features, supported only English, and filtered out low-confidence predictions, limiting coverage. The taxonomy itself contains more than 5,500 hierarchical categories, making the classification problem more complex than a flat label set.

Approach / What changed

The model combines textual product fields with product images. Raw features are converted into embeddings using Multilingual BERT for text and MobileNet-V2 for images, then passed through a seven-level multi-task, multi-class classifier based on the Google Product Taxonomy. Parent outputs influence child predictions, losses from all levels are weighted together, and level-specific confidence thresholds filter low-confidence results.

Takeaways

• The categorization model uses both text and visual features rather than relying only on product text.
• Seven taxonomy levels are modeled as separate multi-class tasks, with parent predictions feeding subsequent child-level predictions.
• The upgraded model increased leaf precision by 8% while almost doubling coverage; thresholds were manually tuned for sensitive categories such as Religious and Ceremonial.

Related reading