Circuit Breakers: The Recovery Loop That Spends Capacity
A circuit breaker turns dependency failures into fast failures, then probes for recovery. See the state loop and how half-open probes can quietly consume worker capacity.
Fly.ioIncident / postmortem
Fly.io responded to CVE-2026-99999, a Linux kernel bug in IPv4 checksum-offload metadata validation, by patching and rebooting its entire fleet over 10 hours on October 16, 2026. The bug allowed a guest to rewrite two bytes of an IPv4 header after the host applied routing and firewall rules, enabling firewall bypass and an out-of-bounds read of up to 32 bytes of host memory. The team split each region into waves so no Managed Postgres cluster had both nodes rebooted simultaneously, with the largest region requiring 17 waves. Some hosts failed to return cleanly, including a chassis in gru that lost its power supply, delaying recovery for those machines until October 20. The post-mortem led to a new Incident Communications Plan that shifts communication decisions from incident responders to the exec team.
CVE-2026-99999 is a Linux kernel bug in IPv4 checksum-offload metadata validation that lets a guest rewrite two bytes of an IPv4 header after the host has already applied routing and firewall rules, enabling firewall bypass and an out-of-bounds read of up to 32 bytes of host memory.
Patched the Linux kernel and rebooted the entire fleet host by host over 10 hours, splitting each region into waves so that no Managed Postgres cluster had both nodes in the same wave.
CVE-2026-99999 exploits a flaw in Linux IPv4 checksum-offload metadata validation, letting a guest rewrite two bytes of an IPv4 header after the host has already applied routing and firewall rules.
The bug enables two attacks: a firewall bypass (rewriting source or destination IP) and an out-of-bounds read (rewriting header length to capture up to 32 bytes of adjacent host memory).
Encrypt traffic between your own services even on a private network, scope agent credentials to the job with short expiration, and treat everything an agent reads as hostile.
The bug lets a guest rewrite two bytes of an IPv4 header after the host has already applied routing and firewall rules. This enables a firewall bypass (rewriting source or destination IP) and an out-of-bounds read (rewriting the header length to capture up to 32 bytes of adjacent host memory).
The host must calculate the checksum in software for the exploit to succeed. The attacker must point the checksum at a different header field, pick a message that produces the desired checksum, and force software checksum calculation.
The team posted an emergency maintenance notice without naming the CVE to avoid confirming vulnerability while the exploit was live. The post-mortem led to a new Incident Communications Plan that puts communication decisions on the exec team instead of the incident responders.
Summaries are generated from the original by a language model. Check the original before relying on it.
See the idea
A circuit breaker turns dependency failures into fast failures, then probes for recovery. See the state loop and how half-open probes can quietly consume worker capacity.
Cilium uses eBPF programs in the Linux kernel to enforce identity-based network policies and capture L7 observability for Kubernetes workloads. This diagram traces the path from packet arrival to policy verdict and Hubble flow.
Keep reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
One useful email. Unsubscribe anytime.