Data-Centric Machine Learning: Building Shopify Inbox’s Message Classification Model

Summary

Shopify Inbox needed to classify millions of monthly customer messages so merchants could prioritize responses and guide product decisions. The team adopted a data-centric approach, improving the dataset rather than primarily changing model architectures. An initial prototype used Universal Sentence Encoder embeddings, UMAP and HDBScan clustering, semi-supervised labeling, and a neural classifier, but reached about 70% accuracy with 35% high-confidence coverage. For version 2.0, the team created a more explicit taxonomy, used human-labeled data and fine-tuned DistilBERT, while addressing privacy and class imbalance through data handling and targeted annotation. In production, version 2.0 achieved about 90% accuracy and 80% high-confidence coverage, although the taxonomy and training data require ongoing updates as conversation trends change.

Context

Shopify Inbox exchanged millions of unique messages per month, making it difficult to understand what customers were asking about. The initial prototype produced only about 70% accuracy and 35% high-confidence coverage because its automatically assisted labels were not always ground truth, and its topics conflated meanings or left many messages uncategorized.

Approach / What changed

The team improved the training data by defining a detailed taxonomy with domain experts, manually annotating messages, and using a pre-trained DistilBERT model. It also used clustering and semi-supervised labeling for the prototype, Presidio and Faker to replace personal information in messages, and model predictions to locate additional examples from underrepresented topics.

Takeaways

• The initial prototype used Universal Sentence Encoder embeddings, UMAP, HDBScan, semi-supervised labeling, and a simple dense neural network, achieving about 70% accuracy and 35% high-confidence coverage.
• The production taxonomy expanded from 20 to 45 classes, while the training set decreased from 40,000 examples based on embedding similarity to 20,000 human-labeled examples.
• Version 2.0 reached about 90% accuracy and 80% high-confidence coverage; the team expects ongoing taxonomy and training-data updates as conversation trends change.

Related reading