Loading post…
HerokuOwen JacobsonIncident / postmortem
How We Found and Fixed a Filesystem Corruption Bug
Summary
An aborted Cedar-14 stack-image upgrade led container tools to truncate and overwrite a disk image while it was mounted as the read-only / filesystem for running dynos. Because direct writes bypassed filesystem caching and consistency mechanisms, reads alternated between cached pages from the old image and missing data from the truncated file, partially corrupting newly loaded programs. Applications consequently experienced elevated segmentation faults and other hard crashes for about five hours; a virtual-machine reproduction using an ext4 loop-mounted image and sleep reliably produced the same failure. Engineers migrated affected dynos to fresh hosts, then changed host-side permissions so installed filesystem images are read-only, and retested the conditions that triggered the incident.
Context
A container-tools upgrade changed the digest algorithms and checksum filenames without considering checksum files from previous versions. During an aborted stack-image rollout, redeploying the original image therefore bypassed the protection against overwriting an existing image, truncating it while running dynos still mounted it as their root filesystem.
Approach / What changed
Engineers identified the filesystem-cache interaction, reproduced the crash with an ext4 loop-mounted disk image and the sleep program, migrated affected dynos to fresh hosts, and marked installed filesystem images read-only on the host filesystem. They retested the conditions that caused the incident.
Takeaways
Directly truncating a mounted disk image bypassed filesystem caching and consistency mechanisms, leaving reads split between cached pages and missing data from the underlying image.
The resulting partially corrupted programs commonly crashed with segmentation faults when execution reached missing or damaged instructions or data.
Migrating dynos to fresh hosts resolved affected filesystems, while making installed images read-only was intended to prevent future overwriting-related failures.
Source and freshness
- Source
- heroku.com
- Published
- Added to Yomu
Summaries are generated from the original by a language model. Check the original before relying on it.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
The best engineering reads, once a week
One useful email. Unsubscribe anytime.