---
title: "Incident Response"
description: "59 posts about Incident Response, summarised, each linking to the original."
---

# Incident Response
> 59 posts about Incident Response, summarised, each linking to the original.

## Articles

### [When the sensor starts thinking: SnortML, agentic AI, and the evolving architecture of intrusion detection](https://yomu.fyi/post/when-the-sensor-starts-thinking-snortml-agentic-ai-and-the-evolving-ar.md)
- Company: [Stack Overflow](https://yomu.fyi/company/stack-overflow.md)
- Author: Samaresh Kumar Singh
- Published: Jul 6, 2026
- Type: Explainer

Classic intrusion detection signatures suffer from an exposure window between zero-day discovery and rule distribution, leaving slight exploit variations unflagged. Cisco Talos addressed this by embedding SnortML into Snort 3 to perform local, sub-millisecond machine learning inference alongside signature checks. SnortML employs an LSTM preceded by a byte-embedding layer and XNNPACK acceleration to score URI queries and POST bodies for vulnerabilities like SQL injection, XSS, and command injection. Running both detection mechanisms concurrently provides independent error profiles and boosts detection confidence when both fire. The broader defense landscape is shifting toward connecting these packet-level ML detectors with agentic AI pipelines and supervised feedback loops.


### [From Chaos to Clarity: How We Built a Unified, Self-Routing Support Ops Ticketing System at Lyft](https://yomu.fyi/post/from-chaos-to-clarity-how-we-built-a-unified-self-routing-support-ops.md)
- Company: [Lyft](https://yomu.fyi/company/lyft.md)
- Author: Atulgupta
- Published: Jun 9, 2026
- Type: Problem & solution

Lyft Urban Solutions' Support Ops team faced fragmented ticketing across four Jira Help Centers with duplicate intake forms, lack of routing logic, and no dashboards. Over five years, the team redesigned the intake architecture by replacing redundant forms with dynamic Jira Proforma forms featuring conditional branching. Automated Jira rules were introduced to assign tickets, apply a taxonomy of over 30 labels, and trigger PagerDuty alerts for P0 incidents. To unify operations across disparate backend projects, the system automatically clones and links tickets from a single portal to downstream project boards. Jira Structures and a Jira-to-Mode ETL pipeline provided cross-project rollups and business analytics before an upcoming Jira Cloud migration required dashboard rebuilds.


### [Your agent can't fix what it can't see](https://yomu.fyi/post/your-agent-can-t-fix-what-it-can-t-see.md)
- Company: [Sentry](https://yomu.fyi/company/sentry.md)
- Author: Sergiy Dybskiy
- Published: May 26, 2026
- Type: Announcement

Coding agents frequently fail to resolve production bugs when restricted to source code alone, as they lack critical runtime context such as stack traces, breadcrumbs, and environment data. To bridge this gap, Sentry introduces Sentry MCP and the Sentry CLI to supply agents and developers with structured production telemetry. The Model Context Protocol integration supports clients like Claude Code, Cursor, and VS Code, while the CLI enables scripted workflows. When an alert triggers, an agent uses trace and commit metadata to diagnose root causes across service boundaries, generate regression tests, and open draft pull requests. Over six months, the Autofix pull request merge rate increased from 41% to 46%.


### [Goodbye limitations, hello data: How Qonto is rethinking observability with ClickHouse Cloud](https://yomu.fyi/post/goodbye-limitations-hello-data-how-qonto-is-rethinking-observability-w.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Author: ClickHouse Team
- Published: May 1, 2026
- Type: Problem & solution

Qonto struggled with slow queries, system instability, and aggressive data sampling when handling distributed traces in Grafana Tempo. To enable wide-event querying without strict cardinality limits, the team migrated their telemetry storage to ClickHouse Cloud. Telemetry from applications, Kubernetes, and GitHub routes through OpenTelemetry collectors over AWS PrivateLink into ClickHouse, while Grafana remains the primary dashboard interface. By compressing 231 TB of uncompressed trace attributes down to 376 GB, Qonto achieved substantial storage savings and integrated an AI incident companion using the ClickHouse Model Context Protocol. The new architecture eliminated cardinality policing, allowed product and support teams to self-serve incident investigations, and enabled real-time streaming pre-aggregations using Apache Flink.


### [How GitHub uses eBPF to improve deployment safety](https://yomu.fyi/post/how-github-uses-ebpf-to-improve-deployment-safety.md)
- Company: [Github](https://yomu.fyi/company/github.md)
- Author: Lawrence Gripper
- Published: Apr 16, 2026
- Type: Problem & solution

Deployment scripts can introduce dangerous circular dependencies when they rely on services or assets from platforms that are currently experiencing outages. Blocking network access at the host level is impractical because stateful nodes continue serving live traffic during rolling deployments. To solve this, GitHub isolates deploy scripts into dedicated Linux cGroups and attaches custom eBPF programs via the cilium/ebpf Go library. The system uses socket-address hooks to redirect DNS queries to a userspace proxy that checks a domain blocklist, while egress packet hooks map DNS transaction IDs to process IDs. This approach successfully prevents deploy-time circular dependencies, provides full command-line audit logs for blocked requests, and speeds up incident recovery.


### [Managing context in long-run agentic applications](https://yomu.fyi/post/managing-context-in-long-run-agentic-applications.md)
- Company: [Slack](https://yomu.fyi/company/slack.md)
- Author: Dominic Marks
- Published: Apr 13, 2026
- Type: Problem & solution

Long-running multi-agent systems struggle with context management because accumulating raw message histories degrades inference quality, increases latency, and exceeds context window limits. In a collaborative security investigation platform, passing unrestricted history can also introduce confirmation bias across specialized agents. To maintain coherence across unbounded rounds of investigation, the system eliminates raw message history carryover between invocations. Instead, it coordinates agents through three structured context channels: a Director's Journal for orchestration memory, a Critic's Review that scores findings to filter hallucinations, and a Critic's Timeline of validated chronological events. This architecture provides agents with tailored context without overwhelming their inference capacity.


### [Navigating Regional Network Blocks](https://yomu.fyi/post/navigating-regional-network-blocks.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: sara\_read
- Published: Mar 26, 2026
- Type: Incident / postmortem

Supabase describes how regional network blocks disrupted access to its multi-tenant developer platform and shares three incidents from September 2025 through March 2026. It explains that DNS-level or transport-layer blocking can affect every tenant when projects share domains and IP addresses, while adding a hosting domain to the Public Suffix List provides only partial mitigation. In the UAE, a project distributing unauthorized streaming content triggered an approximately 18-day block that ended after removal and engagement with the TDRA. Yemen saw TCP connections dropped for about eight days with no official explanation, while India’s roughly eight-day restriction was rescinded after project enforcement and technical engagement with MeitY. The post says providers should build regulator contacts, strengthen abuse response pipelines, and develop alternatives such as proxy domains, custom domains, and other workarounds.


### [Introducing AI observability notebooks for Managed ClickStack in Private Preview](https://yomu.fyi/post/introducing-ai-observability-notebooks-for-managed-clickstack-in-priva.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Author: Mike Shi
- Published: Mar 18, 2026
- Type: Announcement

ClickHouse has launched AI Notebooks for Managed ClickStack in private preview to assist site reliability engineers during system investigations. Instead of isolating interactions in a standard chat window, the platform embeds Anthropic's Claude models directly into a cell-based notebook canvas. The system connects the model to internal ClickStack APIs, translating natural language questions into structured queries that leverage ClickHouse optimizations like materialized views and query chunking. Engineers can inspect reasoning, edit generated SQL queries, add manual visualizations, and branch investigations to evaluate multiple hypotheses independently. While currently utilizing a last-write-wins model for shared canvases, the roadmap includes true concurrent editing, customizable context, and external tool integration via an MCP server.


### [Build agents that run automatically](https://yomu.fyi/post/build-agents-that-run-automatically.md)
- Company: [Cursor](https://yomu.fyi/company/cursor.md)
- Author: Jack Pertschuk, Jon Kaplan & Josh Ma
- Published: Mar 5, 2026
- Type: Announcement

Cursor Automations introduces always-on agents that run on schedules or react to events from Slack, Linear, GitHub, PagerDuty, or custom webhooks. Each invocation spins up a cloud sandbox, uses configured models and MCPs, verifies its output, and can draw on memory from past runs. The product supports review and monitoring workflows, including security audits on pushes to main, risk-based codeowner assignment, and PagerDuty-driven incident investigation that proposes fixes. Other automations generate weekly repository digests, add tests after merged code, triage Slack bug reports through Linear, and assemble cross-tool dashboards. Examples from Cursor and Rippling report faster incident response and reduced repetitive work, while Runlayer describes a software factory built with Runlayer MCP and plugins.


### [Supabase incident on February 12, 2026](https://yomu.fyi/post/supabase-incident-on-february-12-2026.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: paul\_copplestone
- Published: Feb 13, 2026
- Type: Incident / postmortem

Supabase reports a 3-hour-and-42-minute outage on February 12–13, 2026, affecting every service in its us-east-2 region, including Postgres, Auth, Data APIs, Edge Functions, Storage, and Realtime. A deployment of an internal monitoring service enabled AWS VPC Block Public Access in block-bidirectional mode, blocking internet gateway traffic across production VPCs in the region. The incident response was delayed because alerts pointed toward shared services, the relevant CloudTrail event was inconspicuous, pre-production lacked us-east-2, and infrastructure specialists were not involved initially. Supabase restored connectivity by rolling back and removing the monitoring stack, then audited regions, added AWS Organizations policies and IaC safeguards, and planned account isolation, external connectivity probes, production-parity testing, and improved escalation.


### [Engineering at scale, with purpose.](https://yomu.fyi/post/engineering-at-scale-with-purpose.md)
- Company: [Thumbtack](https://yomu.fyi/company/thumbtack.md)
- Author: Thumbtack People Team
- Published: Feb 5, 2026
- Type: Explainer

Senior software engineer Brett Shouse outlines site reliability engineering initiatives and infrastructure modernization efforts underway at Thumbtack. The engineering organization is currently leading an operational project to migrate multiple self-hosted observability services onto a single, unified SaaS platform. This architectural transition consolidates application logs, distributed traces, and system metrics into one accessible view tailored for engineers, customer support staff, and company executives. Moving away from legacy self-hosted monitoring systems reduces systems administration overhead, mitigates alert fatigue, and lowers direct operational infrastructure costs. Concurrently, the reliability team tackles technical debt accumulated from rapid organizational growth by establishing structured incident response processes and automating repetitive operational toil.


### [From Python3.8 to Python3.10: Our Journey Through a Memory Leak](https://yomu.fyi/post/from-python3-8-to-python3-10-our-journey-through-a-memory-leak.md)
- Company: [Lyft](https://yomu.fyi/company/lyft.md)
- Author: Jay Patel
- Published: Dec 15, 2025
- Type: Incident / postmortem

During an initiative to upgrade Python services from version 3.8 to 3.10, Lyft engineers encountered severe latency spikes and timeouts in one service within their test environment. Profiling DynamoDB queries revealed that gevent thread joins were taking up to 30 seconds while pod memory consumption climbed steadily. To isolate the issue, engineers used an internal tracemalloc-based profiler triggered via SIGUSR2 signals, temporarily disabling Gunicorn preload to prevent workers from terminating prematurely on signal receipt. Memory traces pointed to botocore and an incompatibility between weakref.finalize and gevent monkey patching in urllib3 version 1.26.16, which prevented connections from returning to the pool. Downgrading urllib3 to version 1.26.15 immediately resolved both the timeouts and the memory leak before a permanent fix arrived in gevent and urllib3 updates.


### [Streamlining Security Investigations with Agents](https://yomu.fyi/post/streamlining-security-investigations-with-agents.md)
- Company: [Slack](https://yomu.fyi/company/slack.md)
- Author: Dominic Marks
- Published: Dec 1, 2025
- Type: Problem & solution

Slack's Security Engineering team handles billions of daily security events and needed a reliable way to streamline on-call alert triage. An initial prototype relying on a single 300-word prompt produced inconsistent results and frequently reached spurious conclusions without properly challenging assumptions. To gain precise control, the team decomposed the workflow into chained model invocations with structured JSON outputs organized across three agent personas: a Director, four domain experts, and a Critic. Domain experts gather raw evidence through tool calls, the Critic evaluates finding quality and synthesizes a timeline, and the Director steers investigation phases using tiered model costs. The multi-agent system enables engineers to supervise investigations via a real-time dashboard while uncovering emergent issues like credential exposures across process ancestry chains.


### [Kurt Got Got](https://yomu.fyi/post/kurt-got-got.md)
- Company: [Fly.io](https://yomu.fyi/company/fly-io.md)
- Published: Oct 8, 2025
- Type: Incident / postmortem

Fly.io recounts the takeover of its X/Twitter account after CEO Kurt Mackey entered credentials on members-x.com in response to a phishing email from alerts-x.com. The attackers revoked existing tokens, configured new 2FA, changed the account email, and used it for a crypto scam; the company says no customer systems or accounts were at risk, though the incident caused more than 15 hours of brand damage. Fly.io attributes the exposure to treating the shared social account as a legacy exception rather than protecting it through its normal Google IdP and phishing-resistant MFA. The post explains how FIDO2, U2F, and Passkeys prevent fake sites from receiving real credentials, while noting that the 1Password browser plugin could also have flagged the mismatched domain. The account is now protected with Passkeys.


### [Charting a New Course for SaaS Security: Why MongoDB Helped Build the SSCF](https://yomu.fyi/post/charting-a-new-course-for-saas-security-why-mongodb-helped-build-the-s.md)
- Company: [MongoDB](https://yomu.fyi/company/mongodb.md)
- Author: Boris Sieklik
- Published: Sep 30, 2025
- Type: Problem & solution

Widespread enterprise SaaS adoption has introduced notable security blind spots due to inconsistent controls, visibility, and configurations across disparate vendor applications. While traditional compliance frameworks such as SOC 2 and ISO 27001 evaluate provider organizational security, they typically fail to address customer-facing application capabilities. To resolve this discrepancy, MongoDB collaborated with the Cloud Security Alliance and GuidePoint Security to develop the SaaS Security Capability Framework. This framework establishes actionable technical controls across six critical operational domains, including identity and access management, machine-readable logging, configuration management, and incident notifications. By standardizing customer-side capabilities within the shared responsibility model, the framework simplifies vendor risk assessments and provides engineering teams with structured implementation benchmarks.


### [Taming A Voracious Rust Proxy](https://yomu.fyi/post/taming-a-voracious-rust-proxy.md)
- Company: [Fly.io](https://yomu.fyi/company/fly-io.md)
- Published: Feb 26, 2025
- Type: Incident / postmortem

Fly edge proxies began reporting elevated HTTP errors and sharply higher CPU utilization on several IAD hosts, with restarting fly-proxy providing only temporary relief. Profiling an affected process showed Rust tracing and repeated Future polls dominating samples while little time was spent in the kernel, pointing to an async busy-loop. Investigation narrowed the fully qualified polling type to a stack containing the proxy’s Duplex and Rustls’s TlsStream, whose mishandled Waker behavior after a TLS CloseNotify could spin when buffered data remained underneath. Tigris Data’s load test, involving tens of thousands of connections that sent small HTTP bodies and terminated early, triggered the condition; stopping the test and deploying the fix ended the spin-outs. The incident prompted plans to instrument spurious wakeups and reinforced that dependency updates require deliberate tracking, testing, and prioritization.


### [Canva incident report: API Gateway outage](https://yomu.fyi/post/canva-incident-report-api-gateway-outage.md)
- Company: [Canva](https://yomu.fyi/company/canva.md)
- Author: Brendan Humphreys
- Published: Dec 20, 2024
- Type: Incident / postmortem

On November 12, 2024, canva.com suffered an outage lasting approximately one hour due to cascading API Gateway failures. The disruption began when a stale Cloudflare routing rule induced severe latency and packet loss between Ashburn and Singapore during an editor asset deployment. Cloudflare's concurrent streaming mechanism consolidated over 270,000 user requests for a single delayed JavaScript chunk; when the asset finally loaded, clients simultaneously generated a 1.5 million requests-per-second thundering herd to Canva's API Gateway. This surge collided with a telemetry performance regression involving lock contention on Netty event loop threads, exhausting off-heap memory and causing the Linux Out Of Memory Killer to terminate API Gateway containers. Canva mitigated the failure by temporarily blocking all traffic at the CDN level, stabilizing replacement Amazon ECS tasks, and gradually restoring traffic under strict rate limits.


### [Simple error handling strategies with Stripe Workbench](https://yomu.fyi/post/simple-error-handling-strategies-with-stripe-workbench.md)
- Company: [Stripe](https://yomu.fyi/company/stripe.md)
- Author: Ben Smith
- Published: Sep 2, 2024
- Type: Tutorial

Stripe Workbench addresses the difficulty of finding and understanding errors produced by Stripe API invocations. It gives developers a view of recent errors, groups repeated failures, shows their frequency, and links summaries to request logs and actionable resolution guidance. The article describes Stripe error categories such as StripeCardError, StripeInvalidRequestError, and StripeConnectionError, then demonstrates Node.js try/catch handling that branches on the error type. It also covers webhook responses for payment\_intent.payment\_failed events, including retrieving event.data.object.last\_payment\_error, and explains how objects such as PaymentIntents, Setup Intents, Invoices, Setup Attempts, Payouts, and Refunds retain failure information for later inspection. Combined, these mechanisms support targeted responses, troubleshooting through detailed request and response data, and more reliable Stripe integrations.


### [How to Categorize and Prevent Risks of Sensitive Links in URLScan](https://yomu.fyi/post/how-to-categorize-and-prevent-risks-of-sensitive-links-in-urlscan.md)
- Company: [Tinder](https://yomu.fyi/company/tinder.md)
- Author: Tinder
- Published: Nov 7, 2022
- Type: Problem & solution

Security integrations with URLScan can accidentally index sensitive links when API calls are executed with public visibility. Tinder Security Labs constructed targeted queries to uncover exposed links across file-sharing services, enterprise ticketing platforms, and single sign-on portals. Investigations revealed that exposed links included document magic links containing contracts and bank details, as well as active new-hire registration links that allowed attackers to complete registration workflows and configure two-factor authentication. In one SSO query alone, researchers discovered 1,300 indexed invitation links, with 20 to 50 new links appearing daily. The researchers collaborated with URLScan to purge the exposed indexes from public searches and advised organizations to configure security scanning tools to use private or unlisted visibility settings.


### [Hubble: Our Tool for Encapsulating and Extending Security Tools](https://yomu.fyi/post/hubble-our-tool-for-encapsulating-and-extending-security-tools.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Oct 21, 2022
- Type: Explainer

Shopify’s Trust team faced rising complexity in securing a distributed device fleet, including many tools, provisioning and patching work, varied networks, and the loss of in-person remediation. Hubble was built as a unified layer over mobile device management and other security systems, ingesting and standardizing their data while sending commands back with granular access controls and centralized auditing. IT staff use it for inventory, device management, and security, while employees can view device health and compliance, receive remediation guidance, and manage test-device or beta participation. The platform also serves as a standardized source for automation, and the post presents encapsulation and cross-team investment as ways to reduce administrative overhead, support proactive security, and limit risk.


[Newer posts](https://yomu.fyi/topic/incident-response.md) · [Older posts](https://yomu.fyi/topic/incident-response/page/3.md)