---
title: "Spin Infrastructure Adventures: Containers, Systemd, and CGroups"
description: "Spin’s infrastructure team investigated instance instability after Kubernetes nodes experienced frequent failures, including roughly five nodes per day, and instances showed memory use that could exceed their assigned limits. Spin runs instances as Kubernetes pods, with systemd inside each container managing initialization and processes, and the team compared resource behavior across kubectl exec and Spin shell sessions. Stress tests showed that processes started through kubectl exec entered the kubepods cgroup hierarchy, while processes launched through Spin shell entered a hierarchy with an effectively unlimited memory limit, causing usage to appear on the node rather than the pod. Experiments comparing containerd, Docker, and Podman identified systemd cgroup delegation as the relevant difference: Podman isolated systemd and its child processes correctly, while the production setup used cgroupfs. The team identified switching COS to the systemd cgroup driver as the forthcoming fix, while noting that the bind-mounted cgroup filesystem enabled the escape."
---

# Spin Infrastructure Adventures: Containers, Systemd, and CGroups

[Shopify](https://yomu.fyi/company/shopify) · 2023-10-18 · Jul 15, 2022

**Type:** Incident / postmortem

## Summary

Spin’s infrastructure team investigated instance instability after Kubernetes nodes experienced frequent failures, including roughly five nodes per day, and instances showed memory use that could exceed their assigned limits. Spin runs instances as Kubernetes pods, with systemd inside each container managing initialization and processes, and the team compared resource behavior across kubectl exec and Spin shell sessions. Stress tests showed that processes started through kubectl exec entered the kubepods cgroup hierarchy, while processes launched through Spin shell entered a hierarchy with an effectively unlimited memory limit, causing usage to appear on the node rather than the pod. Experiments comparing containerd, Docker, and Podman identified systemd cgroup delegation as the relevant difference: Podman isolated systemd and its child processes correctly, while the production setup used cgroupfs. The team identified switching COS to the systemd cgroup driver as the forthcoming fix, while noting that the bind-mounted cgroup filesystem enabled the escape.

## Context

Kubernetes nodes experienced instability and frequent replacement, while some Spin instances were OOM-killed despite showing memory usage below their pod limits. Later investigation found that processes launched through Spin shell could use memory outside the Kubernetes pod’s resource-limited cgroup hierarchy, allowing instances to interfere with one another.

## Approach / What changed

The team isolated the issue with a Spin instance on its own node, monitored pod and node usage with kubectl top, and used stress to generate controlled memory load. They compared cgroup placement for processes launched through kubectl exec and Spin shell, reproduced the behavior with crictl, ctr, and local Docker, and compared Docker, containerd, and Podman systemd integration. The identified fix was using the systemd cgroup driver instead of cgroupfs; COS version 101 was expected to switch its default.

## Takeaways

- Processes launched through kubectl exec were placed in the kubepods hierarchy and honored pod memory limits, while processes launched through Spin shell used a hierarchy with an effectively unlimited limit.
- Podman’s systemd integration placed both systemd and its child processes in a container-specific cgroup, allowing resource limits to be delegated correctly; Docker showed the cgroup leakage.
- Switching the host from Ubuntu to Container Optimized OS reduced OOM kills by 100 times, but a separate cgroup delegation issue remained and was later traced to the cgroup driver and bind-mounted cgroup filesystem.

**Tags:** [Docker](https://yomu.fyi/topic/docker), [Kubernetes](https://yomu.fyi/topic/kubernetes), [Performance](https://yomu.fyi/topic/performance), [Reliability](https://yomu.fyi/topic/reliability)

- Source: [Shopify](https://shopify.engineering/spin-infrastructure-adventures-containers-systemd-cgroups)
- Source URL: https://shopify.engineering/spin-infrastructure-adventures-containers-systemd-cgroups
- Ingested by Yomu: 2026-08-30T13:40:21.647Z

[Read original post](https://shopify.engineering/spin-infrastructure-adventures-containers-systemd-cgroups)