Loading…
How Databricks Uses AI to Accelerate Incident Investigation
Avijeet Gupta, Bhuban Seth, Kusum Madarasu
- Source
- Databricks
- Published
- Added to Yomu
Summary
Databricks describes AI SRE, an AI-powered debugging agent for operating hundreds of microservices across more than 1,500 Kubernetes clusters, 70+ regions, and three clouds. It addresses incident investigations by assembling context from platform health, service telemetry, changes, dependencies, and team-specific runbooks, with automatic triage followed by interactive, natural-language investigation. The platform uses layered primitives, purpose-built APIs, and a core orchestration engine; team-owned agentic runbooks encode domain procedures, while controlled access and evidence links support auditing. The post reports support for more than 150 teams, 250+ weekly active users, and over 2,000 daily investigations, with users saving several hours of debugging time.
Context
On-call engineers had to connect signals across dashboards, logs, deployments, dependencies, cloud infrastructure, and runbooks largely in their own minds. Interviews found that context assembly consumed 60–80% of investigation time, operational knowledge was unevenly distributed, and application-layer debugging could miss broader platform failures.
Approach / What changed
AI SRE begins automatic triage when an incident fires, running platform health checks, service-level analysis, and team-specific agentic runbooks in parallel. It provides a layered platform with operational-data primitives, purpose-built APIs for normalized and controlled access, and interactive natural-language investigation. Recommendations include traceable evidence, while guardrails manage agent access and activity.
Takeaways
- AI SRE’s automatic triage runs platform health checks, service analysis, and runbook execution in parallel before the on-call engineer starts investigating.
- Teams can convert existing runbooks into agentic runbooks using skills that draw on code, observability data, and past incident history.
- The system emphasizes traceable evidence and graceful degradation: when confidence is insufficient, it presents gathered evidence without asserting an unsupported root cause.