Streamlining Security Investigations with Agents

Summary

Slack's Security Engineering team handles billions of daily security events and needed a reliable way to streamline on-call alert triage. An initial prototype relying on a single 300-word prompt produced inconsistent results and frequently reached spurious conclusions without properly challenging assumptions. To gain precise control, the team decomposed the workflow into chained model invocations with structured JSON outputs organized across three agent personas: a Director, four domain experts, and a Critic. Domain experts gather raw evidence through tool calls, the Critic evaluates finding quality and synthesizes a timeline, and the Director steers investigation phases using tiered model costs. The multi-agent system enables engineers to supervise investigations via a real-time dashboard while uncovering emergent issues like credential exposures across process ancestry chains.

Takeaways

1. 1

   Single-prompt systems lacked fine-grained control for complex investigations, requiring Slack to transition to chained model invocations constrained by JSON schemas.

2. 2

   Slack structured agent tasks into a knowledge pyramid, strategically allocating low, medium, and high-cost models to experts, the critic, and the director.

3. 3

   A weakly adversarial Critic agent reviews tool calls and assigns credibility scores to expert findings, mitigating hallucinations and catching issues missed by domain experts.