How Shopify improved consumer search intent with real-time ML

Summary

Shopify describes how it built near-real-time text and image embedding pipelines to support semantic search, replacing keyword matching with representations that enable similarity-based retrieval. Using Google Cloud’s Dataflow and Apache Beam, the pipelines process roughly 2,500 embeddings per second, preprocess images, run GPU-backed inference, and write outputs to both a data warehouse and an event topic for downstream ingestion. The engineering discussion focuses on trade-offs among memory, throughput, latency, and cost: reducing harness threads cut memory use about 2.6x and allowed a return to cheaper workers, while model sharing reduced throughput. For batching, bursty input produced bundles of one, and stateful batching was rejected because its shuffle increased latency; Dataflow’s per-process model instances instead provided enough parallelism to saturate GPUs.

Context

Shopify wanted merchant product and image updates to become available on storefronts instantly, while maintaining relevant search and interactive shopping experiences. Its streaming embedding pipelines also had to balance cost, throughput, latency, GPU utilization, and worker memory.

Approach / What changed

Shopify uses Google Cloud Dataflow and Apache Beam to run near-real-time text and image inference. Events trigger preprocessing and embedding generation, after which vectors are written to BigQuery for offline analysis and an output topic for real-time Storefront Search ingestion. The implementation tunes worker harness threads, model loading, and batching to manage memory and GPU throughput.

Takeaways

• Reducing Dataflow’s worker harness threads from 12 to 4 lowered the image pipeline’s memory footprint from near 104 GB to about 40 GB, a roughly 2.6x decrease, and removed the need for higher-memory workers that cost 14% more.
• Sharing one embedding model across worker processes substantially reduced GPU memory consumption but degraded throughput, so Shopify retained Dataflow’s default configuration with one model instance per process.
• Stateful batching could provide larger GPU batches but required a shuffle and added too much latency for this pipeline; in-bundle batching was retained while the team continued investigating alternatives.

Related reading