---
title: "Debugging Systems in the Cloud: MySQL, Kubernetes, and Cgroups"
description: "KateSQL, Shopify’s Database-as-a-Service platform on GKE, investigated why otherwise comparable MySQL Pods took between 10 and 30 minutes to be created during instance replacement. Measurements found extreme differences for Pods with identical resources and datasets: one required 2,120 seconds for initialization and 1,104 seconds for startup, while another took 74 and 17 seconds, respectively; affected Pods also accumulated more slow queries involving temporary memory tables. Profiling showed InnoDB’s single-threaded buffer pool initialization spending substantial time in mmap calls and the Linux memory cgroup function mem_cgroup_commit_charge, while affected nodes had roughly 50,000 memory cgroups. The team linked the slowdown to a Linux kernel cgroup memory leak and slab reparenting behavior, then improved performance through node replacement, memory.force_empty testing, and GKE 1.18 upgrades with relevant fixes."
---

# Debugging Systems in the Cloud: MySQL, Kubernetes, and Cgroups

[Shopify](https://yomu.fyi/company/shopify) · 2023-10-18 · Oct 21, 2021

**Type:** Incident / postmortem

## Summary

KateSQL, Shopify’s Database-as-a-Service platform on GKE, investigated why otherwise comparable MySQL Pods took between 10 and 30 minutes to be created during instance replacement. Measurements found extreme differences for Pods with identical resources and datasets: one required 2,120 seconds for initialization and 1,104 seconds for startup, while another took 74 and 17 seconds, respectively; affected Pods also accumulated more slow queries involving temporary memory tables. Profiling showed InnoDB’s single-threaded buffer pool initialization spending substantial time in mmap calls and the Linux memory cgroup function mem\_cgroup\_commit\_charge, while affected nodes had roughly 50,000 memory cgroups. The team linked the slowdown to a Linux kernel cgroup memory leak and slab reparenting behavior, then improved performance through node replacement, memory.force\_empty testing, and GKE 1.18 upgrades with relevant fixes.

## Context

During KateSQL instance replacement, MySQL Pod creation times varied from 10 to 30 minutes, impairing the ability to replace instances quickly for maintenance tasks such as configuration changes and upgrades. Slow Pods also showed gradually declining performance and more slow queries involving temporary memory tables. Newer Kubernetes nodes performed better than nodes with months of uptime, despite matching software versions.

## Approach / What changed

The team compared initialization and startup timings, analyzed MySQL and InnoDB behavior, captured strace and on-CPU perf data, examined slab allocation and memory cgroup counts, and consulted Linux and Google kernel engineers. They mitigated the issue by moving Pods off older nodes, tested memory.force\_empty to remove dead cgroups, evaluated cache-dropping and workload-isolation workarounds, and created GKE 1.18 clusters with COS and upstream cgroup fixes.

## Takeaways

- Identical MySQL Pods showed sharply different timings: one took 2,120 seconds to initialize and 1,104 seconds to start, while another took 74 and 17 seconds.
- Affected nodes had approximately 50,000 memory cgroups, and perf profiling showed InnoDB initialization spending much of its time in mem\_cgroup\_commit\_charge.
- GKE 1.18 clusters produced consistent InnoDB buffer pool initialization and query performance after several weeks of operation.

**Tags:** [Kubernetes](https://yomu.fyi/topic/kubernetes), [Monitoring](https://yomu.fyi/topic/monitoring), [MySQL](https://yomu.fyi/topic/mysql), [Performance](https://yomu.fyi/topic/performance), [Reliability](https://yomu.fyi/topic/reliability)

- Source: [Shopify](https://shopify.engineering/debugging-systems-cloud-mysql-kubernetes-cgroups)
- Source URL: https://shopify.engineering/debugging-systems-cloud-mysql-kubernetes-cgroups
- Ingested by Yomu: 2026-08-30T15:28:51.954Z

[Read original post](https://shopify.engineering/debugging-systems-cloud-mysql-kubernetes-cgroups)