# Incident Response > 10 posts about Incident Response, summarised, each linking to the original. ## Articles ### [How Agentforce-Powered AI Security Workflows Accelerate Incident Response](https://yomu.fyi/post/how-agentforce-powered-ai-security-workflows-accelerate-incident-respo.md) - Company: [Salesforce](https://yomu.fyi/company/salesforce.md) - Author: Scott Nyberg - Published: Aug 12, 2026 Salesforce engineers expanded Security Center from a basic conversational interface into an Agentforce-powered stateful investigations platform for incident response. Managing enterprise security incidents required handling long-running lifecycles, disparate telemetry formats, and finite LLM context windows that risk reasoning pocket collapse. To prevent overflowing context limits, the architecture partitions telemetry into contextual segments and uses AI-driven summarization pipelines before feeding data to the agent. Because non-deterministic LLM outputs break traditional deterministic testing, the team implemented AI-driven evaluation pipelines using simulated customer interactions to evaluate response behavior rather than exact wording. Ongoing work focuses on grounding public foundation models with curated, Salesforce-owned security knowledge to improve platform-specific remediation guidance. ### [How GitHub uses eBPF to improve deployment safety](https://yomu.fyi/post/how-github-uses-ebpf-to-improve-deployment-safety.md) - Company: [Github](https://yomu.fyi/company/github.md) - Author: Lawrence Gripper - Published: Apr 16, 2026 Deployment scripts can introduce dangerous circular dependencies when they rely on services or assets from platforms that are currently experiencing outages. Blocking network access at the host level is impractical because stateful nodes continue serving live traffic during rolling deployments. To solve this, GitHub isolates deploy scripts into dedicated Linux cGroups and attaches custom eBPF programs via the cilium/ebpf Go library. The system uses socket-address hooks to redirect DNS queries to a userspace proxy that checks a domain blocklist, while egress packet hooks map DNS transaction IDs to process IDs. This approach successfully prevents deploy-time circular dependencies, provides full command-line audit logs for blocked requests, and speeds up incident recovery. ### [Managing context in long-run agentic applications](https://yomu.fyi/post/managing-context-in-long-run-agentic-applications.md) - Company: [Slack](https://yomu.fyi/company/slack.md) - Author: Dominic Marks - Published: Apr 13, 2026 Long-running multi-agent systems struggle with context management because accumulating raw message histories degrades inference quality, increases latency, and exceeds context window limits. In a collaborative security investigation platform, passing unrestricted history can also introduce confirmation bias across specialized agents. To maintain coherence across unbounded rounds of investigation, the system eliminates raw message history carryover between invocations. Instead, it coordinates agents through three structured context channels: a Director's Journal for orchestration memory, a Critic's Review that scores findings to filter hallucinations, and a Critic's Timeline of validated chronological events. This architecture provides agents with tailored context without overwhelming their inference capacity. ### [Streamlining Security Investigations with Agents](https://yomu.fyi/post/streamlining-security-investigations-with-agents.md) - Company: [Slack](https://yomu.fyi/company/slack.md) - Author: Dominic Marks - Published: Dec 1, 2025 Slack's Security Engineering team handles billions of daily security events and needed a reliable way to streamline on-call alert triage. An initial prototype relying on a single 300-word prompt produced inconsistent results and frequently reached spurious conclusions without properly challenging assumptions. To gain precise control, the team decomposed the workflow into chained model invocations with structured JSON outputs organized across three agent personas: a Director, four domain experts, and a Critic. Domain experts gather raw evidence through tool calls, the Critic evaluates finding quality and synthesizes a timeline, and the Director steers investigation phases using tiered model costs. The multi-agent system enables engineers to supervise investigations via a real-time dashboard while uncovering emergent issues like credential exposures across process ancestry chains. ### [Improving Hugo stability and addressing oncall challenges through automation](https://yomu.fyi/post/improving-hugo-stability-and-addressing-oncall-challenges-through-auto.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Shuguang Xiang - Published: Mar 20, 2025 Grab's Hugo platform manages over 4,000 data ingestion pipelines for the company's data lake, but recurring failures caused significant on-call strain and data downtime due to complex triage processes. To resolve these operational challenges, an automated architecture was introduced featuring signal collection, automated diagnosis, root-cause tracking, auto-resolution, and a central dashboard. Rather than parsing extensive execution logs from Spark or Airflow, the diagnosis engine analyzes direct computation signals and temporal execution steps in parallel to pinpoint assignees and root causes. An asynchronous auto-resolution framework executes custom recovery handlers, such as automated backoff retries during database replica lags. This automated workflow reduced on-call workloads, improved dataset visibility, and shortened issue triage times across Grab's data ecosystem. ### [Uncovering the Truth Behind Lua and Redis Data Consistency](https://yomu.fyi/post/uncovering-the-truth-behind-lua-and-redis-data-consistency.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Allen Wang - Published: Sep 7, 2020 Grab experienced replica CPU usage spikes following service deployments in their master/replica Redis cluster, which caused failovers to spike to 100% CPU. Investigation revealed that a post-deployment Lua monitor script executed separately on both nodes and relied on non-deterministic HGETALL key ordering. Redis encodes hash objects as either ziplists or hashtables, and restoring from an RDB snapshot initializes small hashes as ziplists even if the master previously converted them to hashtables. This encoding discrepancy caused key ordering to diverge, preventing secondary data from deleting correctly and bloating dataset sizes. Grab resolved the issue by sorting the outputs of HKEYS and HGETALL within the Lua script to guarantee deterministic execution across nodes. ### [Preventing Pipeline Calls from Crashing Redis Clusters](https://yomu.fyi/post/preventing-pipeline-calls-from-crashing-redis-clusters.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Michael Cartmell - Published: May 5, 2019 A single Redis slave node failure caused Grab's Apollo booking service to suffer an over 95 percent call failure rate for one minute despite running a three-shard cluster with two replicas per partition. Investigation revealed that the service configured Go-Redis to route all read queries exclusively to slave nodes to offload master CPU usage. When the slave node dropped offline, batched HMGET pipeline calls failed completely because the client wrapper treated a single command failure as a failure of the entire pipeline. Furthermore, the Go-Redis client cached cluster topology and only lazily refreshed state every sixty seconds, continuing to direct traffic to the dead replica until the timer expired. Grab addressed this risk by recommending dedicated pipeline clients configured with latency-based routing to allow reads to fall back to responsive master nodes. ### [Deep Dive into Database Timeouts in Rails](https://yomu.fyi/post/deep-dive-into-database-timeouts-in-rails.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Jia Hao Goh - Published: Jan 29, 2018 Following a production outage where a database failover caused a Ruby on Rails application to exhaust its Puma server threads, an investigation was conducted to understand how ActiveRecord and MySQL timeout settings behave. A reproduction environment using Docker, Puma, and Toxiproxy replicated how hanging requests to a failing database consume all available server threads, ultimately starving unrelated endpoints. The analysis breaks down ActiveRecord connection pooling mechanics alongside underlying mysql2 and libmysqlclient settings, specifically checkout\_timeout, connect\_timeout, and read\_timeout. Testing confirmed how existing and new TCP connections transition through socket states during network interruptions while waiting on configured timeout intervals. ### [Dealing with the Meltdown Patch at Grab](https://yomu.fyi/post/dealing-with-the-meltdown-patch-at-grab.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Althaf Hameez - Published: Jan 7, 2018 AWS infrastructure maintenance related to Meltdown patches led to severe CPU utilization spikes across Grab's ElastiCache Redis instances. Because Redis is single-threaded, spikes past 50% CPU on two-vCPU instances threatened service capacity, and initial Multi-AZ failovers only provided temporary relief until the new master nodes received rolling patches. To handle the increased overhead before their peak traffic window, the engineering team horizontally scaled both clustered and non-clustered Redis fleets. For Redis 3.2.4 clusters lacking live re-sharding support, they provisioned larger clusters, warmed caches, and redirected traffic. Non-clustered workloads were resolved by provisioning extra nodes, migrating compatible services to Redis Cluster, or updating application code to shard data across multiple instances. ### [Grab You Some Post-Mortem Reports](https://yomu.fyi/post/grab-you-some-post-mortem-reports.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Lian Yuanlin - Published: Feb 4, 2016 Grab uses a Service-Oriented Architecture to deploy features quickly, but unfamiliarity across teams makes cross-service production debugging difficult. Historical incident reports lacked context, diagnostic details, impact data, and timelines, leaving outside engineers unable to learn from past outages. To address this, Grab established a four-pillar framework for post-mortem reports covering chronology, context, empowerment, and solutions. The approach mandates blameless, educational write-ups that categorize post-incident improvements across people, product, and process dimensions. Final reports undergo peer reviews by engineers from external teams to ensure clarity and remove bias.