---
title: "Leveraging Go Worker Pools to Scale Server-side Data Sharing"
description: "Shopify’s Server Pixels service validates, processes, augments, and produces more than one billion customer events daily, but rising traffic caused Kafka input-topic consumption lag when downstream processing slowed. Its original design could spawn an unlimited number of goroutines while processing or producing event batches, consuming CPU and memory and threatening the five-minute event-processing SLO. The service was redesigned around fixed-size Go worker pools: workers receive Job values through channels, process them via reusable work functions, and use wait groups and context cancellation for orderly shutdown. Load testing showed 15 workers in both processor and producer raised throughput to 12.9 thousand events per second per pod, while 50 workers reached 19.3 thousand; subsequent improvements reached 21 thousand, a 170% increase over 7.75 thousand, and the service handled a maximum of 46 thousand events per second during BFCM 2021."
---

# Leveraging Go Worker Pools to Scale Server-side Data Sharing

[Shopify](https://yomu.fyi/company/shopify) · 2023-10-18 · Sep 9, 2022

**Type:** Problem & solution

## Summary

Shopify’s Server Pixels service validates, processes, augments, and produces more than one billion customer events daily, but rising traffic caused Kafka input-topic consumption lag when downstream processing slowed. Its original design could spawn an unlimited number of goroutines while processing or producing event batches, consuming CPU and memory and threatening the five-minute event-processing SLO. The service was redesigned around fixed-size Go worker pools: workers receive Job values through channels, process them via reusable work functions, and use wait groups and context cancellation for orderly shutdown. Load testing showed 15 workers in both processor and producer raised throughput to 12.9 thousand events per second per pod, while 50 workers reached 19.3 thousand; subsequent improvements reached 21 thousand, a 170% increase over 7.75 thousand, and the service handled a maximum of 46 thousand events per second during BFCM 2021.

## Context

Rising customer-event volume caused Kafka input-topic consumption lag when downstream components slowed. The original design spawned an unlimited number of goroutines for batch processing and production, increasing CPU and memory use and risking violation of the five-minute event-processing SLO. The service also needed to prepare for three times the usual load during BFCM.

## Approach / What changed

The service adopted fixed-size Go worker pools. A shared Worker interface accepts jobs through Go channels and invokes a configurable workFunc, allowing processor, producer, validation, parsing, and augmentation components to use the same pattern. Wait groups track workers during shutdown, while context cancellation triggers remaining batches to be sent, channel closure, and orderly completion of in-flight jobs.

## Takeaways

- The Worker interface and configurable workFunc let different pipeline components process distinct Job types while reusing the same worker-pool structure.
- Shutdown coordination uses context cancellation, channel closure, and wait groups so in-flight batches finish before workers and the output channel are closed.
- Raising the worker count from 15 to 50 increased measured throughput from 12.9 to 19.3 thousand events per second per pod; later improvements reached 21 thousand.

**Tags:** [Go](https://yomu.fyi/topic/go), [Kafka](https://yomu.fyi/topic/kafka), [Performance](https://yomu.fyi/topic/performance), [Scalability](https://yomu.fyi/topic/scalability)

- Source: [Shopify](https://shopify.engineering/leveraging-go-worker-pools)
- Source URL: https://shopify.engineering/leveraging-go-worker-pools
- Ingested by Yomu: 2026-08-30T13:39:09.329Z

[Read original post](https://shopify.engineering/leveraging-go-worker-pools)