---
title: "Capturing Every Change From Shopify’s Sharded Monolith"
description: "Shopify’s separate internal warehouse and Merchant Analytics Platform created duplicated work, differing extraction results, and incompatible tooling while demand grew for fresher, low-latency merchant data. Longboat, its query-based CDC service, polls sharded-monolith replicas in batch jobs using updated_at, but queries can run only hourly and cannot detect hard deletes or updates that bypass that field. Shopify therefore adopted Kafka and Debezium CDC to capture database changes, standardize extraction, and make the event stream usable by both streaming and batch consumers. CDC-backed Beam modeling reduced average freshness for one migration from one day to one hour, while Kafka became a shared source for operational and analytical data. Shopify still identifies incremental snapshotting and streaming-state problems, including reconciling MySQL table chunks with ongoing binlog events without expensive locks, as ongoing work."
---

# Capturing Every Change From Shopify’s Sharded Monolith

[Shopify](https://yomu.fyi/company/shopify) · 2023-10-18 · Mar 12, 2021

**Type:** Problem & solution

## Summary

Shopify’s separate internal warehouse and Merchant Analytics Platform created duplicated work, differing extraction results, and incompatible tooling while demand grew for fresher, low-latency merchant data. Longboat, its query-based CDC service, polls sharded-monolith replicas in batch jobs using updated\_at, but queries can run only hourly and cannot detect hard deletes or updates that bypass that field. Shopify therefore adopted Kafka and Debezium CDC to capture database changes, standardize extraction, and make the event stream usable by both streaming and batch consumers. CDC-backed Beam modeling reduced average freshness for one migration from one day to one hour, while Kafka became a shared source for operational and analytical data. Shopify still identifies incremental snapshotting and streaming-state problems, including reconciling MySQL table chunks with ongoing binlog events without expensive locks, as ongoing work.

## Context

Shopify’s warehouse and Merchant Analytics Platform used separate extraction tools, standards, and results, causing duplicated work as merchant analytics demands expanded. The batch-oriented Longboat system also limited freshness and could miss hard deletes or changes that did not update updated\_at.

## Approach / What changed

Shopify adopted Kafka and Debezium for change-data capture from the sharded monolith, using CDC events as a shared source for streaming and batch consumers. The platform also uses Beam modeling and is investigating incremental snapshots that reconcile MySQL query results with ongoing binlog events without locking tables.

## Takeaways

- Longboat queries individual tables through read-only replicas and can run a given table query no more than once an hour because of source-database performance impact.
- Query-based extraction captures soft deletes through updated\_at but misses hard deletes, because the deleted row no longer exists to be fetched.
- CDC-backed Beam modeling reduced average freshness for one migration from one day to one hour, while ongoing work includes incremental snapshots and streaming joins or materialized state.

**Tags:** [Data Pipelines](https://yomu.fyi/topic/data-pipelines), [Kafka](https://yomu.fyi/topic/kafka), [MySQL](https://yomu.fyi/topic/mysql), [Streaming](https://yomu.fyi/topic/streaming)

- Source: [Shopify](https://shopify.engineering/capturing-every-change-shopify-sharded-monolith)
- Source URL: https://shopify.engineering/capturing-every-change-shopify-sharded-monolith
- Ingested by Yomu: 2026-08-31T01:10:04.153Z

[Read original post](https://shopify.engineering/capturing-every-change-shopify-sharded-monolith)