How Shopify improved consumer search intent with real-time ML

Summary

Shopify describes how it built near-real-time text and image embedding pipelines to support semantic search, replacing keyword matching with representations that enable similarity-based retrieval. Using Google Cloud’s Dataflow and Apache Beam, the pipelines process roughly 2,500 embeddings per second, preprocess images, run GPU-backed inference, and write outputs to both a data warehouse and an event topic for downstream ingestion. The engineering discussion focuses on trade-offs among memory, throughput, latency, and cost: reducing harness threads cut memory use about 2.6x and allowed a return to cheaper workers, while model sharing reduced throughput. For batching, bursty input produced bundles of one, and stateful batching was rejected because its shuffle increased latency; Dataflow’s per-process model instances instead provided enough parallelism to saturate GPUs.

Takeaways

1. 1

   Reducing Dataflow’s worker harness threads from 12 to 4 lowered the image pipeline’s memory footprint from near 104 GB to about 40 GB, a roughly 2.6x decrease, and removed the need for higher-memory workers that cost 14% more.

2. 2

   Sharing one embedding model across worker processes substantially reduced GPU memory consumption but degraded throughput, so Shopify retained Dataflow’s default configuration with one model instance per process.

3. 3

   Stateful batching could provide larger GPU batches but required a shuffle and added too much latency for this pipeline; in-bundle batching was retained while the team continued investigating alternatives.