---
title: "Using Server Sent Events to Simplify Real-time Streaming at Scale"
description: "Shopify’s 2022 BFCM Live Map needed to deliver real-time sales and product data to thousands of concurrent users while processing data from millions of merchants. Its 2021 architecture used WebSocket delivery to a presentation layer, which stored messages in a mailbox that clients polled at least every 10 seconds, creating latency and bottlenecks. Shopify replaced that path with a streamlined Flink-based pipeline and a Golang SSE server that subscribes to Kafka topics and pushes JSON data to registered, authenticated clients as soon as it is available. The SSE server ran horizontally behind Nginx load balancers, with a configurable Java client used to load-test concurrent connections. The 2022 system maintained 100 percent uptime, delivered SSE data within milliseconds of availability, and visualized data within 21 seconds of creation, including pipeline processing."
---

# Using Server Sent Events to Simplify Real-time Streaming at Scale

[Shopify](https://yomu.fyi/company/shopify) · 2023-10-18 · Nov 30, 2022

**Type:** Problem & solution

## Summary

Shopify’s 2022 BFCM Live Map needed to deliver real-time sales and product data to thousands of concurrent users while processing data from millions of merchants. Its 2021 architecture used WebSocket delivery to a presentation layer, which stored messages in a mailbox that clients polled at least every 10 seconds, creating latency and bottlenecks. Shopify replaced that path with a streamlined Flink-based pipeline and a Golang SSE server that subscribes to Kafka topics and pushes JSON data to registered, authenticated clients as soon as it is available. The SSE server ran horizontally behind Nginx load balancers, with a configurable Java client used to load-test concurrent connections. The 2022 system maintained 100 percent uptime, delivered SSE data within milliseconds of availability, and visualized data within 21 seconds of creation, including pipeline processing.

## Context

The 2021 BFCM Live Map used WebSocket delivery followed by a presentation-layer mailbox and client polling, taking at least 10 seconds and sometimes minutes for trending-product changes to reach users. Shopify needed a communication model and pipeline that could handle millions of orders, thousands of concurrent connections, peak BFCM volume, and lower data latency.

## Approach / What changed

Shopify replaced the presentation-layer polling path and deprecated Cricket in favor of a Flink-based pipeline backed by its Trickle streaming platform. A Golang SSE server subscribed to Kafka topics and pushed JSON payloads to registered authenticated clients. The server was deployed horizontally behind Nginx load balancers, while a configurable Java client simulated high volumes of concurrent SSE connections for capacity testing.

## Takeaways

- SSE fit the Live Map because it provided server-to-client delivery over HTTP, automatic reconnection, and a read-only stream without requiring WebSocket’s bidirectional channel.
- The SSE server subscribed to Kafka topics and pushed data to all registered client connections as soon as it became available, removing the mailbox storage and polling delay.
- Load testing used a configurable Java client and okhttp-eventsource to simulate many concurrent connections and determine how many users each server pod could handle.

**Tags:** [Go](https://yomu.fyi/topic/go), [Kafka](https://yomu.fyi/topic/kafka), [Performance](https://yomu.fyi/topic/performance), [Scalability](https://yomu.fyi/topic/scalability), [Streaming](https://yomu.fyi/topic/streaming)

- Source: [Shopify](https://shopify.engineering/server-sent-events-data-streaming)
- Source URL: https://shopify.engineering/server-sent-events-data-streaming
- Ingested by Yomu: 2026-08-30T13:37:16.466Z

[Read original post](https://shopify.engineering/server-sent-events-data-streaming)