Debugging Systems in the Cloud: MySQL, Kubernetes, and Cgroups

Summary

KateSQL, Shopify’s Database-as-a-Service platform on GKE, investigated why otherwise comparable MySQL Pods took between 10 and 30 minutes to be created during instance replacement. Measurements found extreme differences for Pods with identical resources and datasets: one required 2,120 seconds for initialization and 1,104 seconds for startup, while another took 74 and 17 seconds, respectively; affected Pods also accumulated more slow queries involving temporary memory tables. Profiling showed InnoDB’s single-threaded buffer pool initialization spending substantial time in mmap calls and the Linux memory cgroup function mem_cgroup_commit_charge, while affected nodes had roughly 50,000 memory cgroups. The team linked the slowdown to a Linux kernel cgroup memory leak and slab reparenting behavior, then improved performance through node replacement, memory.force_empty testing, and GKE 1.18 upgrades with relevant fixes.

Takeaways

1. 1

   Identical MySQL Pods showed sharply different timings: one took 2,120 seconds to initialize and 1,104 seconds to start, while another took 74 and 17 seconds.

2. 2

   Affected nodes had approximately 50,000 memory cgroups, and perf profiling showed InnoDB initialization spending much of its time in mem_cgroup_commit_charge.

3. 3

   GKE 1.18 clusters produced consistent InnoDB buffer pool initialization and query performance after several weeks of operation.