Loading…
Browse by type
Incident / postmortem reads
Incident / postmortem posts on Yomu, summarised newest first.
- 46 posts
- 20 companies
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Incident / postmortem posts
Latency Incident Report: Socket Bind Before Connect
An August 11, 2016 incident caused increased routing latency in Heroku’s EU Common Runtime, even though CPU, packet, memory, request, and socket metrics remained normal. The routing layer received EADDRINUSE while connecting to dynos, leading engineers to investigate Linux TCP port allocation, TIME_WAIT, and the Erlang VM’s system calls with strace. They found that the VM called bind() with 0.0.0.0:0 before connect(), forcing Linux to allocate an ephemeral source port without knowing the destination; this made the ephemeral port range an effective per-instance traffic limit and could fail before 65,536 connections. Testing reproduced the failures and showed that setting SO_REUSEADDR before bind(), exposed in Erlang as {reuseaddr, true}, removed the bind errors; the corresponding Vegur patch was deployed a few days later.
A Key Expired in Redis, You Won't Believe What Happened Next
Grab experienced an issue where its Unicorn API served stale data for up to 45 to 60 minutes despite expected cache invalidation times totaling around 11 minutes. The setup utilized ElastiCache Redis 2.x configured with a single master node for writes and two read-only slaves handling reads. Investigation revealed that in Redis 2.x, slave nodes do not expire keys on their own and only delete them upon receiving an explicit DEL command from the master. Because the master only actively checks and deletes 200 random keys per second, clearing expired keys across roughly 5.6 million cached items mathematically required over 110 hours, resulting in slaves serving expired data.
How We Found and Fixed a Filesystem Corruption Bug
An aborted Cedar-14 stack-image upgrade led container tools to truncate and overwrite a disk image while it was mounted as the read-only / filesystem for running dynos. Because direct writes bypassed filesystem caching and consistency mechanisms, reads alternated between cached pages from the old image and missing data from the truncated file, partially corrupting newly loaded programs. Applications consequently experienced elevated segmentation faults and other hard crashes for about five hours; a virtual-machine reproduction using an ext4 loop-mounted image and sleep reliably produced the same failure. Engineers migrated affected dynos to fresh hosts, then changed host-side permissions so installed filesystem images are read-only, and retested the conditions that triggered the incident.
Pulling the Thread on Kafka’s Compacted Topics
Apache Kafka’s log cleaner for compacted topics could crash silently when its offset map lacked enough space to hold a full log segment. Because the broker continued running without that internal thread, superseded data accumulated, threatening node and potentially cluster stability. The post explains compaction’s two phases: building an offset map for newer messages, then rewriting segments while retaining the latest offset for each key, with the latest segment excluded from the map. Kafka was changed to remove the crashing assertion and track partial segment loads so cleaning can recover without fitting an entire segment in memory. The roughly 100-line patch shipped in Kafka 0.10.1.1, which the post identifies as Heroku’s default version, and prevents the cleaner from dying silently.
How We Sped up SNI TLS Handshakes by 5x
Heroku’s SNI TLS termination work exposed severe handshake performance problems under benchmarks designed to avoid session resumption, caching, and keep-alive reuse. Initial tests made nodes slow and unresponsive 7–8x earlier than comparable HTTP tests, while perf top showed Erlang garbage collection and ETS hash operations consuming unexpectedly large amounts of CPU. Tracing linked repeated ets.select_delete calls to the centralized ssl_manager process, where ssl_pkix_db’s certificate cache path was still used even for DER certificates supplied directly in memory. Engineers patched the SSL library to bypass that cache wholesale. Each instance then handled three times the benchmark capacity, while SNI latencies fell from 100–120 milliseconds to 20–25 milliseconds for the reported app subset; the patch was later shipped in Erlang/OTP-19.1.
Preparing Our Response to the OpenSSL Security Flaw
Heroku prepared for an announced high-severity OpenSSL vulnerability whose details were embargoed until patch release. The team began by mapping exposure and system criticality, considering SSL clients, possible man-in-the-middle attacks, encrypted database connections, and worst-case remote code execution, then created stack-image update actions and priorities. Engineers were assigned responsibilities under the existing incident-response process, with communication rules, escalation criteria, paging, and availability planned in advance. When the OpenSSL update and vulnerability details arrived on July 9 at about 8am Central, triage showed it was neither an RCE nor server-side issue, and Ubuntu reported that neither LTS version was affected; because Heroku production used LTS, no remediation or all-hands response was needed. The preparation still exercised exposure review and response coordination for future incidents.
Top topics
- Architecture830
- Performance812
- LLMs648
- Machine Learning617
- Open Source561
- AI Agents542
- AI454
- Developer Experience446
Weekly digest
The best engineering reads, once a week.