# Streamlining Security Investigations with Agents

[Slack](https://yomu.fyi/company/slack) · Dominic Marks · Dec 1, 2025

**Type:** Problem & solution

## Summary

Slack's Security Engineering team handles billions of daily security events and needed a reliable way to streamline on-call alert triage. An initial prototype relying on a single 300-word prompt produced inconsistent results and frequently reached spurious conclusions without properly challenging assumptions. To gain precise control, the team decomposed the workflow into chained model invocations with structured JSON outputs organized across three agent personas: a Director, four domain experts, and a Critic. Domain experts gather raw evidence through tool calls, the Critic evaluates finding quality and synthesizes a timeline, and the Director steers investigation phases using tiered model costs. The multi-agent system enables engineers to supervise investigations via a real-time dashboard while uncovering emergent issues like credential exposures across process ancestry chains.

## Context

Slack's Security Engineering team ingests billions of security events daily and is responsible for reviewing detection alerts during on-call shifts. An initial investigation prototype based on a single 300-word prompt and an MCP server exhibited variable performance, often reaching convenient or spurious conclusions without validating evidence across data sources.

## Approach / What changed

The team separated the investigation workflow into chained model invocations governed by structured JSON schema outputs. They implemented a multi-agent system featuring a Director agent that coordinates phases, four domain expert agents (Access, Cloud, Code, Threat) that query data sources via tool calls, and a Critic agent that scores finding credibility and constructs an investigation timeline.

## Takeaways

- Single-prompt systems lacked fine-grained control for complex investigations, requiring Slack to transition to chained model invocations constrained by JSON schemas.
- Slack structured agent tasks into a knowledge pyramid, strategically allocating low, medium, and high-cost models to experts, the critic, and the director.
- A weakly adversarial Critic agent reviews tool calls and assigns credibility scores to expert findings, mitigating hallucinations and catching issues missed by domain experts.

**Tags:** [Architecture](https://yomu.fyi/topic/architecture), [Incident Response](https://yomu.fyi/topic/incident-response), [LLMs](https://yomu.fyi/topic/llm)

[Read original post](https://slack.engineering/streamlining-security-investigations-with-agents)