Loading…
Topic
59 posts about Incident Response. Every summary links to the original.
See the idea
An architecture view of a bug bounty program: scope, triage, validation, containment, durable fix, and payout, plus the loop that turns each report into stronger defaults.
AIOps automates the detect-investigate-analyze-recommend-execute-verify cycle by combining AI reasoning with infrastructure automation, keeping humans in control of credentials and final decisions.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
The post distills lessons from five years working on Shopify’s payment infrastructure into guidance for building systems that tolerate failure and large-scale traffic. It frames the advice partly around onboarding developers who were new to payments or Shopify’s scale, and notes that load testing for BFCM 2021 began in July with globally distributed load generation. It recommends low timeouts across HTTP clients and data stores, circuit breakers such as Semian, and capacity reasoning with Little’s Law, rate limiting, and load shedding. Monitoring should cover latency, traffic, errors, and saturation, while structured, machine-readable logs make centralized records searchable; load tests use benchmark gateways that mimic production partner latency and capacity. For incidents, Shopify uses defined roles, Slack-based coordination, and retrospectives that turn incorrect assumptions into safeguards and training material.
On September 30th, 2021, Shopify nearly suffered a platform-wide outage after Let’s Encrypt’s old root certificate expired, exposing stale certificate data in internal clients rather than at the public edge. At 10:15 a.m., monitoring detected certificate errors after deployment-related connection changes, and investigation traced them to Ruby services that depended directly or indirectly on httpclient, whose embedded cacert.pem was six years old and did not trust the applicable certificate chain. Shopify forked the library, replaced cacert.pem with a current root certificate bundle, deployed it to canaries, and automated pull requests for affected repositories. The temporary fix restored the monolith, while systems were added to track root-certificate changes and update the fork automatically. Deployment locking kept most monolith pods from cycling, limiting the incident to canaries and causing secondary effects including delayed order confirmations, throttled marketing campaigns, and missing product recommendations.
Shopify’s preparation for Black Friday Cyber Monday (BFCM) centers on building and maintaining resiliency practices for a high-traffic platform and large development teams. Regular load and stress tests verify required capacity, probe limits, detect regressions, and compare behavior across differently configured shops. Teams document expected user experiences and dependency failure modes in a resiliency matrix, then use game days to test that model against reality, including alerts, paging, operator readiness, and response procedures. Incident analysis treats disruptions as learning opportunities rather than stopping at a single “root cause”; nearer to BFCM, Shopify slows feature and service changes while increasing work on performance, reliability, scale, runbooks, tools, and bottlenecks. The source says these practices support a platform that powered over $5.1 billion in sales.
Incident #2105 began during a production deployment of a Go-based Git server after a configuration variable had been set but not transmitted to production instances. Staging deployment completed successfully before the same change was attempted in production. The deployment sent a new code version to one batch, whose unauthenticated calls to a private API produced HTTP 401 responses that were returned publicly; the author paused before deploying the remaining batches and resolved the issue by synchronizing configuration, killing affected processes, and allowing them to restart. The post argues that the failure was enabled by a process requiring separate, infrequent manual actions rather than being solely an individual omission. Its proposed remedy is to combine configuration synchronization with code deployment, or alternatively warn when deployment begins with unsynchronized variables, supported by a blameless retrospective.
Grab experienced replica CPU usage spikes following service deployments in their master/replica Redis cluster, which caused failovers to spike to 100% CPU. Investigation revealed that a post-deployment Lua monitor script executed separately on both nodes and relied on non-deterministic HGETALL key ordering. Redis encodes hash objects as either ziplists or hashtables, and restoring from an RDB snapshot initializes small hashes as ziplists even if the master previously converted them to hashtables. This encoding discrepancy caused key ordering to diverge, preventing secondary data from deleting correctly and bloating dataset sizes. Grab resolved the issue by sorting the outputs of HKEYS and HGETALL within the Lua script to guarantee deterministic execution across nodes.
Shopify’s Point Of Sale (POS) app is mission-critical for brick-and-mortar merchants, because downtime can create long lineups, unhappy customers, and lost sales. Unlike server-side systems, mobile fixes depend on Apple and Google review, and merchants may delay installing approved updates, so the team focuses on preventing bugs and shortening recovery time. Its practices include layered automated and manual testing, component ownership with automatic Code Owners reviews, automated mobile “tophatting,” and weekly releases supported by a rotating Release Captain and ShipIt Mobile. Staged rollouts, beta flags, minimum-version enforcement for rare critical security fixes, and dedicated Primary and Secondary ATCs limit exposure and clarify incident response. ChatOps-based incident channels capture findings for root cause analysis, while rotating on-call responsibilities reduce context switching; these measures helped the app scale to larger merchants while shipping faster and improving reliability.
An unexpected viral submission on Hacker News prompted sudden alpha traffic for Supabase, attracting 30,000 visitors and over 1,400 signups in one week. The surge strained initial infrastructure, which initially relied on a single server running Docker Compose for middleware and Digital Ocean for customer databases. System bottlenecks emerged rapidly, including hit limits on Digital Ocean droplet counts, frequent physical server faults, and a three-hour provisioning outage when Cloudflare reached its 1,000-subdomain limit. In response, the team migrated middleware to multiple globally load-balanced ECS clusters via AWS Global Accelerator. Supabase also transitioned 1,800 servers from Digital Ocean to AWS t3a instances, nearly halving database startup times from 90 seconds to 50 seconds.
Game Day tests deliberately trigger failure modes in production systems and examine whether automated behavior and human response match expectations. Shopify frames the practice as preparation for outages that can arise despite code checks, reviews, and other safeguards. The method starts by inventorying possible failures, then selecting scenarios by impact and simulation difficulty rather than estimated likelihood. Each scenario becomes a controlled experiment with prerequisites, testable hypotheses, expected alerts and downstream behavior, plus recovery steps recorded in a spreadsheet. During execution, teams track results, communication, and work distribution; afterward they fix gaps, review coordination, update response guidelines, and rerun tests to verify changes, with periodic testing keeping systems and people prepared.
A single Redis slave node failure caused Grab's Apollo booking service to suffer an over 95 percent call failure rate for one minute despite running a three-shard cluster with two replicas per partition. Investigation revealed that the service configured Go-Redis to route all read queries exclusively to slave nodes to offload master CPU usage. When the slave node dropped offline, batched HMGET pipeline calls failed completely because the client wrapper treated a single command failure as a failure of the entire pipeline. Furthermore, the Go-Redis client cached cluster topology and only lazily refreshed state every sixty seconds, continuing to direct traffic to the dead replica until the timer expired. Grab addressed this risk by recommending dedicated pipeline clients configured with latency-based routing to allow reads to fall back to responsive master nodes.
Heroku frames security as an engineering discipline that must challenge happy-path assumptions, test continuously, and prepare for low-predictability, high-consequence “black swan” events. Its threat-modeling guidance focuses on challenging defaults, recognizing that attackers have resources and organizational support, and planning for unexpected failures such as a zero-day exploit. A control-plane instrument for Heroku Postgres detected privilege escalation to a superuser role, and the Data team isolated the instance within 17 minutes while investigating the activity. The event was traced to researcher Andrew Krasichkov (“buglloc”), whose testing exposed a vulnerability in the dblink and postgres_fdw extensions. Through the Bugcrowd bug bounty program, Heroku worked with him to understand the issue, reward the finding, and defend against the previously unknown flaw before a patch was available.
Applications can fail when a single upstream service goes down, prompting tools that shed load, fail gracefully, and protect struggling dependencies. The discussion frames this as application resilience for everyday failures, distinct from request-level retries, rollback strategies, and catastrophic disaster recovery. It presents maintenance mode as an easily toggled hard stop with a clear message and status-page link, read-only mode as a way to preserve access when modifications fail, and feature flags as controls for global or meaningful groups. Implementation examples include environment variables, relational databases, Redis, in-memory caches, and an S3-based check, with state stored primarily in Postgres and Redis at Heroku. The closing guidance says maintenance mode should be easy to toggle, while read-only mode can preserve user confidence and access to information.
Shopify’s incident management process separates coordination from technical remediation: the incident manager on call (IMOC) leads response, while component experts fix production issues and a Support Response Manager handles public communication. The process follows a simplified Incident Command System hierarchy and moves from detection and confirmation through communication, mitigation, verification, documentation, investigation, and root cause analysis. Its ChatOps chatbot, Spy, runs in Slack through Lita and integrates with PagerDuty, StatusPage, and GitHub via webhooks, providing commands for paging, incident startup, status communication, mitigation, and stopping incidents. Spy binds incidents to a shared #war-room, records notes and event times, generates service disruption documents, sends status-page reminders, and can summon the IMOC squad when an incident lasts beyond a specified period. The result is a more consistent response that reduces context switching and post-incident toil while supporting detailed post-mortems.
Following a production outage where a database failover caused a Ruby on Rails application to exhaust its Puma server threads, an investigation was conducted to understand how ActiveRecord and MySQL timeout settings behave. A reproduction environment using Docker, Puma, and Toxiproxy replicated how hanging requests to a failing database consume all available server threads, ultimately starving unrelated endpoints. The analysis breaks down ActiveRecord connection pooling mechanics alongside underlying mysql2 and libmysqlclient settings, specifically checkout_timeout, connect_timeout, and read_timeout. Testing confirmed how existing and new TCP connections transition through socket states during network interruptions while waiting on configured timeout intervals.
AWS infrastructure maintenance related to Meltdown patches led to severe CPU utilization spikes across Grab's ElastiCache Redis instances. Because Redis is single-threaded, spikes past 50% CPU on two-vCPU instances threatened service capacity, and initial Multi-AZ failovers only provided temporary relief until the new master nodes received rolling patches. To handle the increased overhead before their peak traffic window, the engineering team horizontally scaled both clustered and non-clustered Redis fleets. For Redis 3.2.4 clusters lacking live re-sharding support, they provisioned larger clusters, warmed caches, and redirected traffic. Non-clustered workloads were resolved by provisioning extra nodes, migrating compatible services to Redis Cluster, or updating application code to shard data across multiple instances.
An aborted Cedar-14 stack-image upgrade led container tools to truncate and overwrite a disk image while it was mounted as the read-only / filesystem for running dynos. Because direct writes bypassed filesystem caching and consistency mechanisms, reads alternated between cached pages from the old image and missing data from the truncated file, partially corrupting newly loaded programs. Applications consequently experienced elevated segmentation faults and other hard crashes for about five hours; a virtual-machine reproduction using an ext4 loop-mounted image and sleep reliably produced the same failure. Engineers migrated affected dynos to fresh hosts, then changed host-side permissions so installed filesystem images are read-only, and retested the conditions that triggered the incident.
Heroku’s help application depends on the Platform API for authentication, authorization, and user data, so it must remain available to agents and customers during platform incidents. The open-sourced api-maintenance-sim app tests this requirement by imitating an outage in which every request receives an HTTP 503 Service Unavailable response. After deploying the simulator, teams point the application under test to its hostname through PLATFORM_API_URL, passed to platform-api’s connect_oauth method; when unset, the gem uses the production URI. A heroku config:set command enables the alternate endpoint, making regular simulations possible and helping reveal what failure means for the application or service when the Platform API is unavailable, supporting preparation for incidents and communication during outages.
Grab describes how its service-oriented architecture makes cross-team production debugging difficult because engineers may lack familiarity with other teams’ code and architecture. Post-mortem reports are presented as the glue for shared understanding, but a sample incident entry lacks service context, bug detail, impact, chronology, and investigation steps. The proposed standard has four requirements—Chronology, Context, Empowerment, and Solutions—covering timelines, service internals, blameless educational diagnosis, and remedies across People, Product, and Process. A template includes initial symptoms, a timestamped timeline, log-based investigation with code evidence, temporary and permanent fixes, and improvement actions, while peer review by another team is required before finalization.
Heroku prepared for an announced high-severity OpenSSL vulnerability whose details were embargoed until patch release. The team began by mapping exposure and system criticality, considering SSL clients, possible man-in-the-middle attacks, encrypted database connections, and worst-case remote code execution, then created stack-image update actions and priorities. Engineers were assigned responsibilities under the existing incident-response process, with communication rules, escalation criteria, paging, and availability planned in advance. When the OpenSSL update and vulnerability details arrived on July 9 at about 8am Central, triage showed it was neither an RCE nor server-side issue, and Ubuntu reported that neither LTS version was affected; because Heroku production used LTS, no remediation or all-hands response was needed. The preparation still exercised exposure review and response coordination for future incidents.
Retrospectives are presented as a way for software engineering teams to review operational incidents, investigate root causes, create remediation tasks, and improve teamwork. Beyond technical review, the process can focus on emotional and human aspects of projects, including interactions and ways to work better together. The suggested timing includes post-incident or post-project sessions, mid-project course corrections, and weekly meetings; a 90-minute Friday session one week after a project ends is described as effective, while next-day and three-week delays are discouraged. An active, neutral facilitator should prepare an agenda and guest list, state a clear purpose, involve everyone, and use one or two activities such as a timeline, five whys, or Start Doing / Stop Doing / Keep Doing brainstorming.
The best engineering reads, once a week.
