Loading…
Browse by type
Incident / postmortem reads
Incident / postmortem posts on Yomu, summarised newest first.
- 46 posts
- 20 companies
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Incident / postmortem posts
We Were Wrong About GPUs
Fly.io’s GPU bet was intended to give internet-facing apps fast AI/ML inference through Fly GPU Machines, which pair a hardware-mapped Nvidia GPU with a Fly Machine. Building them required dedicated GPU servers, Intel Cloud Hypervisor with PCI passthrough, driver work, orchestrator changes, security assessments, and handling large model-weight files. The product has not found product-market fit because most software developers want LLM APIs and services such as OpenAI, Anthropic, Claude, GPT, Replicate, or RunPod rather than raw GPUs, while serious GPU users seek much larger clusters and lightweight users may be too uncertain a market. L40S demand exists, but those GPUs are not driving the core business. Fly.io will keep existing GPU Machines but does not plan a near-term v2, concluding that the bet was worth making without compromising its isolation or developer experience.
Canva incident report: API Gateway outage
On November 12, 2024, canva.com suffered an outage lasting approximately one hour due to cascading API Gateway failures. The disruption began when a stale Cloudflare routing rule induced severe latency and packet loss between Ashburn and Singapore during an editor asset deployment. Cloudflare's concurrent streaming mechanism consolidated over 270,000 user requests for a single delayed JavaScript chunk; when the asset finally loaded, clients simultaneously generated a 1.5 million requests-per-second thundering herd to Canva's API Gateway. This surge collided with a telemetry performance regression involving lock contention on Netty event loop threads, exhausting off-heap memory and causing the Linux Out Of Memory Killer to terminate API Gateway containers. Canva mitigated the failure by temporarily blocking all traffic at the CDN level, stabilizing replacement Amazon ECS tasks, and gradually restoring traffic under strict rate limits.
Pumas, Routers & Keepalives—Oh my!
Heroku’s Router 2.0 beta exposed elevated tail response times and request failures in a subset of Puma applications, especially under 200 milliseconds of server-side lag and 30 requests per second. Investigation compared legacy and Router 2.0 connections, examined TIME_WAIT behavior, and tested Go’s http.Transport idle-connection limit; raising MaxIdleConnsPerHost from the default of 2 to 100 stabilized connection states but did not improve latency. The underlying issue was Puma’s handling of persistent connections, where request balancing could leave work queued behind busy connections; disabling keepalives eliminated the problem, while thread, worker, and max_fast_inline tuning were offered as alternatives. Load tests found the issue unique to Puma, and the update reports that Puma 7.0 fixed the bug, recommending version 7.0.4 or higher.
Spin Infrastructure Adventures: Containers, Systemd, and CGroups
Spin’s infrastructure team investigated instance instability after Kubernetes nodes experienced frequent failures, including roughly five nodes per day, and instances showed memory use that could exceed their assigned limits. Spin runs instances as Kubernetes pods, with systemd inside each container managing initialization and processes, and the team compared resource behavior across kubectl exec and Spin shell sessions. Stress tests showed that processes started through kubectl exec entered the kubepods cgroup hierarchy, while processes launched through Spin shell entered a hierarchy with an effectively unlimited memory limit, causing usage to appear on the node rather than the pod. Experiments comparing containerd, Docker, and Podman identified systemd cgroup delegation as the relevant difference: Podman isolated systemd and its child processes correctly, while the production setup used cgroupfs. The team identified switching COS to the systemd cgroup driver as the forthcoming fix, while noting that the bind-mounted cgroup filesystem enabled the escape.
How to make Rust leak memory (also: how to make it stop)
fly-proxy, a Rust-based proxy, leaked memory because its obsolete custom HoneycombLayer retained child spans under a long-lived parent span. The post follows a request from browser input through DNS, TCP, TLS, HTTP/2, and asynchronous Rust components including an aya-built eBPF program, tokio, hyper, and tracing, before explaining how the leak surfaced. In the custom layer, every closed child span was added to its parent’s collection, so each request consumed a small amount of memory while a high-level async task remained alive. Removing that legacy code fixed the primary problem: memory usage barely exceeded 50MB during a load test, while remaining growth came from queues and latency data retained for processing rather than the identified leak.
The Adventures of Rendezvous in Heroku’s New Architecture
Heroku’s migration of Rendezvous, the bidirectional proxy that connects Heroku Run sessions to one-off dynos, from EC2 Classic to AWS VPCs caused one customer’s app to hit H12 timeouts. The move replaced a single Elastic IP ingress and process-level TLS termination with a Network Load Balancer, whose larger TLS frames exposed a blocking flaw in Rendezvous’s single-threaded Ruby channel loop. That loop used IO.select on the TCP socket and dyno terminal, but OpenSSL::SSL::SSLSocket could retain unread buffered data after a partial read; IO.select saw no TCP bytes and blocked beyond the Heroku Router’s 30-second limit. Engineers reproduced the behavior across two languages, instrumented the instance agent, then fixed the channel with separate threads and corrected socket-reading and IO.select handling. After staging tests, the customer moved back to VPC Rendezvous and confirmed the issue was resolved.
That Old Certificate Expired and Started an Outage. This is What Happened Next
On September 30th, 2021, Shopify nearly suffered a platform-wide outage after Let’s Encrypt’s old root certificate expired, exposing stale certificate data in internal clients rather than at the public edge. At 10:15 a.m., monitoring detected certificate errors after deployment-related connection changes, and investigation traced them to Ruby services that depended directly or indirectly on httpclient, whose embedded cacert.pem was six years old and did not trust the applicable certificate chain. Shopify forked the library, replaced cacert.pem with a current root certificate bundle, deployed it to canaries, and automated pull requests for affected repositories. The temporary fix restored the monolith, while systems were added to track root-certificate changes and update the fork automatically. Deployment locking kept most monolith pods from cycling, limiting the incident to canaries and causing secondary effects including delayed order confirmations, throttled marketing campaigns, and missing product recommendations.
How Safari Ruined My Tuesday
Safari 15 introduced an obscure CSS bug that made pointer-events: none persist in LiveView applications. LiveView uses connected and disconnected modes, and its default app.css had included .phx-disconnected * { pointer-events: none; } to disable interaction while disconnected. In Safari 15, the rule could remain effective after reconnection or class changes, leaving links and buttons unresponsive even though the web inspector reported pointer-events: auto. The workaround removes pointer-events: none from app.css, although LiveView already ignores clicks on Phoenix links and buttons while disconnected; LiveView 0.17 instead uses a different disconnected-state class. The change avoids applying the faulty rule from existing user-land CSS while retaining custom loading-state styling, and the post says a Safari Development Preview build might fix the underlying bug.
Debugging Systems in the Cloud: MySQL, Kubernetes, and Cgroups
KateSQL, Shopify’s Database-as-a-Service platform on GKE, investigated why otherwise comparable MySQL Pods took between 10 and 30 minutes to be created during instance replacement. Measurements found extreme differences for Pods with identical resources and datasets: one required 2,120 seconds for initialization and 1,104 seconds for startup, while another took 74 and 17 seconds, respectively; affected Pods also accumulated more slow queries involving temporary memory tables. Profiling showed InnoDB’s single-threaded buffer pool initialization spending substantial time in mmap calls and the Linux memory cgroup function mem_cgroup_commit_charge, while affected nodes had roughly 50,000 memory cgroups. The team linked the slowdown to a Linux kernel cgroup memory leak and slab reparenting behavior, then improved performance through node replacement, memory.force_empty testing, and GKE 1.18 upgrades with relevant fixes.
Debugging High Latency Due to Context Leaks
Market-Store, Grab's feature store for real-time machine learning features, experienced latency spikes from under 200 milliseconds to 2 seconds as traffic grew. Metrics and logs showed no direct correlation to API issues, but heap profiling with PPROF revealed continuously increasing memory held by child contexts. Further analysis tracked the leak to an update in Grab's open-source Async Library, which switched background contexts to uncancelled task contexts for worker runners. Because parent contexts maintained references to these uncancelled child contexts, the garbage collector could not reclaim their memory. This progressive memory exhaustion directly degraded API latency.
How I Broke `git push heroku main`
Incident #2105 began during a production deployment of a Go-based Git server after a configuration variable had been set but not transmitted to production instances. Staging deployment completed successfully before the same change was attempted in production. The deployment sent a new code version to one batch, whose unauthenticated calls to a private API produced HTTP 401 responses that were returned publicly; the author paused before deploying the remaining batches and resolved the issue by synchronizing configuration, killing affected processes, and allowing them to restart. The post argues that the failure was enabled by a process requiring separate, infrequent manual actions rather than being solely an individual omission. Its proposed remedy is to combine configuration synchronization with code deployment, or alternatively warn when deployment begins with unsynchronized variables, supported by a blameless retrospective.
Uncovering the Truth Behind Lua and Redis Data Consistency
Grab experienced replica CPU usage spikes following service deployments in their master/replica Redis cluster, which caused failovers to spike to 100% CPU. Investigation revealed that a post-deployment Lua monitor script executed separately on both nodes and relied on non-deterministic HGETALL key ordering. Redis encodes hash objects as either ziplists or hashtables, and restoring from an RDB snapshot initializes small hashes as ziplists even if the master previously converted them to hashtables. This encoding discrepancy caused key ordering to diverge, preventing secondary data from deleting correctly and bloating dataset sizes. Grab resolved the issue by sorting the outputs of HKEYS and HGETALL within the Lua script to guarantee deterministic execution across nodes.
Alpha Launch Postmortem
An unexpected viral submission on Hacker News prompted sudden alpha traffic for Supabase, attracting 30,000 visitors and over 1,400 signups in one week. The surge strained initial infrastructure, which initially relied on a single server running Docker Compose for middleware and Digital Ocean for customer databases. System bottlenecks emerged rapidly, including hit limits on Digital Ocean droplet counts, frequent physical server faults, and a three-hour provisioning outage when Cloudflare reached its 1,000-subdomain limit. In response, the team migrated middleware to multiple globally load-balanced ECS clusters via AWS Global Accelerator. Supabase also transitioned 1,800 servers from Digital Ocean to AWS t3a instances, nearly halving database startup times from 90 seconds to 50 seconds.
The Curious Case of the Table-Locking UPDATE Query
The post recounts a PostgreSQL performance incident in which metadata processing stopped as jobs accumulated and an UPDATE targeting a single UUID remained active for days. Diagnostics showed the query consuming more than 80% of database time, while manual execution did not reproduce the delay. A pg_stat_activity and pg_locks query exposed many waiting statements and an AccessExclusiveLock entry whose locktype was tuple, leading the investigation toward PostgreSQL row locking and contention. The eventual cause was threaded code that issued concurrent updates to the same row for each job; removing that concurrency, aborting old backends with pg_cancel_backend, and deploying the change restored processing and began churning through the backlog. A correction notes that the reported lock was on a tuple rather than the table, and that PostgreSQL does not have lock promotion as suggested in the debugging discussion.
Puma 4: Hammering Out H13s—A Debugging Story
Heroku investigated mysterious H13s—“Connection closed” without response—affecting Ruby applications during dyno deployments or scaling, especially at high traffic levels. The failure involved a race between SIGTERM shutting down Puma and the router’s separate system continuing to deliver a few requests, but comparison with other webservers indicated Puma needed to handle the condition. A reproduction used Rack applications that sent SIGTERM to Puma and themselves, plus parallel curl requests, reproducing curl exit code 52 and Heroku H13s. Debugging showed Puma relied on process termination to close accepted sockets; Puma 4 instead closes binder listeners before shutdown, allowing the router to retry connections while preserving responses for in-flight requests.
Preventing Pipeline Calls from Crashing Redis Clusters
A single Redis slave node failure caused Grab's Apollo booking service to suffer an over 95 percent call failure rate for one minute despite running a three-shard cluster with two replicas per partition. Investigation revealed that the service configured Go-Redis to route all read queries exclusively to slave nodes to offload master CPU usage. When the slave node dropped offline, batched HMGET pipeline calls failed completely because the client wrapper treated a single command failure as a failure of the entire pipeline. Furthermore, the Go-Redis client cached cluster topology and only lazily refreshed state every sixty seconds, continuing to direct traffic to the dead replica until the timer expired. Grab addressed this risk by recommending dedicated pipeline clients configured with latency-based routing to allow reads to fall back to responsive master nodes.
Rails Asset Pipeline Directory Traversal Vulnerability (CVE-2018-3760)
All previously released versions of Sprockets, which powers the Rails asset pipeline, contained a directory traversal vulnerability, CVE-2018-3760. Rails applications were vulnerable when production runtime asset compilation was enabled with config.assets.compile = true; the default production setting was false, so apps using the default were not vulnerable. Sprockets handled requests by searching for rendered assets and compiling matches dynamically, and an attacker could exploit that process with a crafted URL based on a known absolute source-asset path to make it appear inside a permitted path. Patches were prepared for Sprockets versions 2, 3, and 4, with remediation via disabling runtime compilation or upgrading to versions 2.12.5, 3.7.2, or 4.0.0.beta8. Heroku also updated its Ruby Buildpack to fail affected builds, notified customers, and warned that compromised applications may require credential rotation.
Ruby CVE-2017-17405: Vulnerability in Ruby’s FTP Implementation
CVE-2017-17405 affected Ruby’s net/ftp implementation, where a filename returned by an FTP server could reach Ruby’s open function and enable command execution. The issue was found while investigating CVE-2017-8817 in libcurl and checking whether Ruby used libcurl or had a separate FTP implementation with comparable wildcard behavior. Reviewing Ruby 2.4.2 source identified the 1,496-line ftp.rb file and the gettextfile method, which called open(localfile, "w") without validating the value. A proof of concept made an FTP client request a file list containing a pipe-prefixed payload; Ruby passed it to the shell, producing remote code execution and creating a file containing id output. Ruby replaced open with File.open, released the fix in 2.4.3, and assigned CVE-2017-17405; affected versions included Ruby 2.2.8 and earlier, 2.3.5 and earlier, 2.4.2 and earlier, and 2.5.0-preview1.
A House of Cards: An Exploration of Security When Building Docker Containers
The post examines how a Docker-based, multi-tenant build environment can expose untrusted code when layered isolation is misconfigured. It traces a setup using AWS CodeBuild, Docker-in-Docker, and per-step build containers, then tests directory traversal through configurable Dockerfile paths before focusing on daemon networking. The vulnerability comes from starting dockerd with tcp://0.0.0.0:2375 and placing containers on a shared default network; build code can reach the daemon, issue Docker commands, mount the host filesystem, and retrieve container-managed AWS credentials. The excerpt reports a reverse shell, access to the CodeBuild filesystem, and temporary credentials whose impact depends on IAM permissions, while noting that the build must remain alive. The stated fix is to remove the TCP listener and retain the mapped Unix socket.
Dealing with the Meltdown Patch at Grab
AWS infrastructure maintenance related to Meltdown patches led to severe CPU utilization spikes across Grab's ElastiCache Redis instances. Because Redis is single-threaded, spikes past 50% CPU on two-vCPU instances threatened service capacity, and initial Multi-AZ failovers only provided temporary relief until the new master nodes received rolling patches. To handle the increased overhead before their peak traffic window, the engineering team horizontally scaled both clustered and non-clustered Redis fleets. For Redis 3.2.4 clusters lacking live re-sharding support, they provisioned larger clusters, warmed caches, and redirected traffic. Non-clustered workloads were resolved by provisioning extra nodes, migrating compatible services to Redis Cluster, or updating application code to shard data across multiple instances.
Top topics
- Architecture830
- Performance812
- LLMs648
- Machine Learning617
- Open Source561
- AI Agents542
- AI454
- Developer Experience446
Weekly digest
The best engineering reads, once a week.