---
title: "Data-Centric Machine Learning: Building Shopify Inbox’s Message Classification Model"
description: "Shopify Inbox needed to classify millions of monthly customer messages so merchants could prioritize responses and guide product decisions. The team adopted a data-centric approach, improving the dataset rather than primarily changing model architectures. An initial prototype used Universal Sentence Encoder embeddings, UMAP and HDBScan clustering, semi-supervised labeling, and a neural classifier, but reached about 70% accuracy with 35% high-confidence coverage. For version 2.0, the team created a more explicit taxonomy, used human-labeled data and fine-tuned DistilBERT, while addressing privacy and class imbalance through data handling and targeted annotation. In production, version 2.0 achieved about 90% accuracy and 80% high-confidence coverage, although the taxonomy and training data require ongoing updates as conversation trends change."
---

# Data-Centric Machine Learning: Building Shopify Inbox’s Message Classification Model

[Shopify](https://yomu.fyi/company/shopify) · 2023-10-18 · Jul 21, 2022

**Type:** Problem & solution

## Summary

Shopify Inbox needed to classify millions of monthly customer messages so merchants could prioritize responses and guide product decisions. The team adopted a data-centric approach, improving the dataset rather than primarily changing model architectures. An initial prototype used Universal Sentence Encoder embeddings, UMAP and HDBScan clustering, semi-supervised labeling, and a neural classifier, but reached about 70% accuracy with 35% high-confidence coverage. For version 2.0, the team created a more explicit taxonomy, used human-labeled data and fine-tuned DistilBERT, while addressing privacy and class imbalance through data handling and targeted annotation. In production, version 2.0 achieved about 90% accuracy and 80% high-confidence coverage, although the taxonomy and training data require ongoing updates as conversation trends change.

## Context

Shopify Inbox exchanged millions of unique messages per month, making it difficult to understand what customers were asking about. The initial prototype produced only about 70% accuracy and 35% high-confidence coverage because its automatically assisted labels were not always ground truth, and its topics conflated meanings or left many messages uncategorized.

## Approach / What changed

The team improved the training data by defining a detailed taxonomy with domain experts, manually annotating messages, and using a pre-trained DistilBERT model. It also used clustering and semi-supervised labeling for the prototype, Presidio and Faker to replace personal information in messages, and model predictions to locate additional examples from underrepresented topics.

## Takeaways

- The initial prototype used Universal Sentence Encoder embeddings, UMAP, HDBScan, semi-supervised labeling, and a simple dense neural network, achieving about 70% accuracy and 35% high-confidence coverage.
- The production taxonomy expanded from 20 to 45 classes, while the training set decreased from 40,000 examples based on embedding similarity to 20,000 human-labeled examples.
- Version 2.0 reached about 90% accuracy and 80% high-confidence coverage; the team expects ongoing taxonomy and training-data updates as conversation trends change.

**Tags:** [Machine Learning](https://yomu.fyi/topic/machine-learning), [Performance](https://yomu.fyi/topic/performance), [Privacy](https://yomu.fyi/topic/privacy)

- Source: [Shopify](https://shopify.engineering/shopify-inbox-message-classification-model)
- Source URL: https://shopify.engineering/shopify-inbox-message-classification-model
- Ingested by Yomu: 2026-08-30T13:40:12.977Z

[Read original post](https://shopify.engineering/shopify-inbox-message-classification-model)