Building a Real-time Buyer Signal Data Pipeline for Shopify Inbox

Summary

Shopify built a real-time buyer-signal pipeline for Shopify Inbox after noting that 70 percent of its conversations involved customers making a purchasing decision. The system surfaces cart actions and order-completion information as chronological conversation events, covering events from up to 14 days before a conversation and post-conversation events while its seven-day lifecycle remains active. It uses Kafka through Monorail and Change Data Capture, Apache Beam on Google Cloud Dataflow, filtering jobs for mission-critical transactional events, and stateful aggregation with timers and a Global Window to retain buyer activity. In an equal-group A/B test, buyer signals increased merchant response rate by two percentage points and attributed conversion rate by 0.7 percentage points, while response time showed no significant change.

Context

Most Shopify Inbox conversations involved customers making a purchasing decision, but merchants needed relevant purchase-intent signals to understand where buyers were in their shopping journeys, answer questions more directly, and prioritize conversations likely to convert.

Approach / What changed

The pipeline ingests cart, checkout, and conversation data from Kafka sources including Monorail and Change Data Capture, filters transactional events, and uses Apache Beam stateful processing, timers, and a Global Window to aggregate buyer activity. Google Cloud Dataflow runs the Beam model, which emits cart and order-completion context through Monorail for downstream delivery as conversation events.

Takeaways

• Change Data Capture uses MySQL binlogs and Debezium to turn data that is not naturally streamed into a stream of events, while Monorail provides structured, versioned Kafka events.
• The aggregation job keys conversation, checkout, and cart inputs by the buyer identifier and retains historical state in a Global Window, supporting access to prior events and an extendable cart lifespan.
• The A/B test found a two-percentage-point increase in response rate and a 0.7-percentage-point increase in attributed conversion rate, with no significant change in response time.

Related reading