Streamlining Security Investigations with Agents

Summary

Slack's Security Engineering team handles billions of daily security events and needed a reliable way to streamline on-call alert triage. An initial prototype relying on a single 300-word prompt produced inconsistent results and frequently reached spurious conclusions without properly challenging assumptions. To gain precise control, the team decomposed the workflow into chained model invocations with structured JSON outputs organized across three agent personas: a Director, four domain experts, and a Critic. Domain experts gather raw evidence through tool calls, the Critic evaluates finding quality and synthesizes a timeline, and the Director steers investigation phases using tiered model costs. The multi-agent system enables engineers to supervise investigations via a real-time dashboard while uncovering emergent issues like credential exposures across process ancestry chains.

Context

Slack's Security Engineering team ingests billions of security events daily and is responsible for reviewing detection alerts during on-call shifts. An initial investigation prototype based on a single 300-word prompt and an MCP server exhibited variable performance, often reaching convenient or spurious conclusions without validating evidence across data sources.

Approach / What changed

The team separated the investigation workflow into chained model invocations governed by structured JSON schema outputs. They implemented a multi-agent system featuring a Director agent that coordinates phases, four domain expert agents (Access, Cloud, Code, Threat) that query data sources via tool calls, and a Critic agent that scores finding credibility and constructs an investigation timeline.

Takeaways

• Single-prompt systems lacked fine-grained control for complex investigations, requiring Slack to transition to chained model invocations constrained by JSON schemas.
• Slack structured agent tasks into a knowledge pyramid, strategically allocating low, medium, and high-cost models to experts, the critic, and the director.
• A weakly adversarial Critic agent reviews tool calls and assigns credibility scores to expert findings, mitigating hallucinations and catching issues missed by domain experts.

Related reading

• Slack · Apr 13, 2026

  Managing context in long-run agentic applications

  Long-running multi-agent systems struggle with context management because accumulating raw message histories degrades inference quality, increases latency, and exceeds context window limits. In a collaborative security investigation platform, passing unrestricted history can also introduce confirmation bias across specialized agents. To maintain coherence across unbounded rounds of investigation, the system eliminates raw message history carryover between invocations. Instead, it coordinates agents through three structured context channels: a Director's Journal for orchestration memory, a Critic's Review that scores findings to filter hallucinations, and a Critic's Timeline of validated chronological events. This architecture provides agents with tailored context without overwhelming their inference capacity.

  Dominic Marks

• Slack · May 5, 2026

  From SSH to REST: A Security-Driven Modernization of Slack’s EMR Data Pipelines

  Slack's data platform accumulated over 700 SSH-based Airflow operators executing jobs directly on AWS Elastic MapReduce clusters, creating broad security surfaces and operational instability. Stateful SSH connections frequently failed when Kubernetes pods restarted, leaving orphaned zombie processes and preventing infrastructure modernization such as migrating EMR clusters to child accounts. To eliminate SSH access, Slack adopted a REST-based architecture using Quarry, a gateway providing unified REST APIs and token authentication across compute engines. For non-Hadoop CLI workloads, Slack leveraged YARN Distributed Shell to execute arbitrary shell scripts stored in Amazon S3 within managed, isolated YARN containers. The phased migration achieved zero downtime across eight data regions, unlocking reliable monitoring, simplified onboarding, and future initiatives like Spark on Kubernetes.

  Mahendran Vasagam

• Grab · Mar 19, 2026

  From firefighting to building: How AI agents restored our team’s core productivity

  Grab's Analytics Data Warehouse team spent roughly 40% of their engineering bandwidth answering repetitive questions, tracing data lineage, and handling basic pipeline enhancement requests across more than 15,000 tables. To eliminate these manual investigative bottlenecks, the team implemented a multi-agent AI architecture using FastAPI, LangGraph, Redis, and PostgreSQL. Incoming requests route through two dedicated pathways: an enhancement pipeline for generating code changes and an investigation pipeline for diagnosing data anomalies. Specialized agents interact with underlying engines like Trino, GitLab, and observability platforms to query data, trace transformations, and check ongoing incidents before synthesizing findings. This system automates the context-gathering process within minutes while maintaining human-in-the-loop review for merge requests and production changes.

  Sneh Agrawal

• Salesforce · Aug 12, 2026

  How Agentforce-Powered AI Security Workflows Accelerate Incident Response

  Salesforce engineers expanded Security Center from a basic conversational interface into an Agentforce-powered stateful investigations platform for incident response. Managing enterprise security incidents required handling long-running lifecycles, disparate telemetry formats, and finite LLM context windows that risk reasoning pocket collapse. To prevent overflowing context limits, the architecture partitions telemetry into contextual segments and uses AI-driven summarization pipelines before feeding data to the agent. Because non-deterministic LLM outputs break traditional deterministic testing, the team implemented AI-driven evaluation pipelines using simulated customer interactions to evaluate response behavior rather than exact wording. Ongoing work focuses on grounding public foundation models with curated, Salesforce-owned security knowledge to improve platform-specific remediation guidance.

  Scott Nyberg