---
title: "Scaling Shopify's BFCM Live Map: An Apache Flink Redesign"
description: "Shopify’s 2021 BFCM live map had to scale from more than 1.7 million merchants while adding product trends and unique shoppers, after the prior Cricket-and-Redis backend showed volume and delivery weaknesses. The redesign placed Apache Flink on the critical path to filter irrelevant checkout events, reducing Cricket’s workload to about one percent of event volume, while cross-region sharding and active-active deployment supported availability. Flink computed new metrics, including 500 product categories ranked by sales-volume change every five minutes, while Cricket retained existing metric computation and deduplication. In load tests, the pipeline handled 1 million checkout events per second, and its jobs ran at 100 percent uptime during BFCM without backpressure or manual intervention. Redis still caused high CPU and memory pressure, producing brief arc-visual errors for a small sample of users, so Shopify plans a simpler Flink-centered design."
---

# Scaling Shopify's BFCM Live Map: An Apache Flink Redesign

[Shopify](https://yomu.fyi/company/shopify) · 2023-10-18 · Dec 10, 2021

**Type:** Problem & solution

## Summary

Shopify’s 2021 BFCM live map had to scale from more than 1.7 million merchants while adding product trends and unique shoppers, after the prior Cricket-and-Redis backend showed volume and delivery weaknesses. The redesign placed Apache Flink on the critical path to filter irrelevant checkout events, reducing Cricket’s workload to about one percent of event volume, while cross-region sharding and active-active deployment supported availability. Flink computed new metrics, including 500 product categories ranked by sales-volume change every five minutes, while Cricket retained existing metric computation and deduplication. In load tests, the pipeline handled 1 million checkout events per second, and its jobs ran at 100 percent uptime during BFCM without backpressure or manual intervention. Redis still caused high CPU and memory pressure, producing brief arc-visual errors for a small sample of users, so Shopify plans a simpler Flink-centered design.

## Context

The live map’s previous Cricket-and-Redis architecture faced scalability concerns as Shopify grew beyond 1.7 million merchants. Cricket risked overload from checkout Kafka traffic, while Redis became a bottleneck as published messages, subscribers, and new metrics increased. The system only needed the latest metric values and could tolerate some loss of sampled arc visuals, but the existing connection to browsers could hang and make those visuals disappear temporarily.

## Approach / What changed

Apache Flink was added to filter irrelevant checkout events before they reached Cricket, allowing Cricket to process about one percent of the event volume for existing metrics. Flink computed new product trends and unique-shopper metrics, with cross-region sharding and active-active deployment for high availability; Cricket handled deduplication and relayed Flink’s results. Shopify also built Spark batch-job fallbacks and plans to move all metric computation into Flink, snapshotting results for a web-tier cache.

## Takeaways

- In load tests, the Flink pipeline operated with 1 million checkout events per second at peak, while Flink jobs ran through BFCM with 100 percent uptime, no backpressure, and no manual intervention.
- The product-trends metric emitted 500 product categories with sales-quantity changes every five minutes; change was calculated from prior one-hour sales quantity divided by the mean of the prior six hours, minus one.
- Redis remained a bottleneck after the redesign: message serving caused high CPU loads and product trends consumed substantial memory, leading to brief arc-visual errors that were mitigated by dropping unnecessary Redis state.

**Tags:** [Kafka](https://yomu.fyi/topic/kafka), [Performance](https://yomu.fyi/topic/performance), [Redis](https://yomu.fyi/topic/redis), [Scalability](https://yomu.fyi/topic/scalability), [Streaming](https://yomu.fyi/topic/streaming)

- Source: [Shopify](https://shopify.engineering/bfcm-live-map-2021-apache-flink-redesign)
- Source URL: https://shopify.engineering/bfcm-live-map-2021-apache-flink-redesign
- Ingested by Yomu: 2026-08-30T15:28:19.503Z

[Read original post](https://shopify.engineering/bfcm-live-map-2021-apache-flink-redesign)