---
title: "RADAR: Catch gray failures with anomaly detection"
description: "Gray failures are partial customer-facing problems that can leave health checks green while a subset of users quietly fails. RADAR (Reliability Anomaly Detection, Alerting, and Root-cause analysis) addresses this gap by collecting error counts and distinct affected users, segmented by error code and region, then applying the unsupervised streaming SPOT model. SPOT learns normal behavior from the previous 14 days using a single risk parameter, while downstream stages enrich, filter, deduplicate, and route alerts before providing anomaly details and AI/BI Genie-backed dashboards for investigation. Running RADAR internally reduced incident-discovery time by 95% with over 90% precision and no human needed to spot the pattern. The post also describes a Databricks scaffold that maps metrics, detection, alerting, and root-cause analysis to platform components and can be assembled by an AI agent from a metric, scaffold, and prompt."
---

# RADAR: Catch gray failures with anomaly detection

[Databricks](https://yomu.fyi/company/databricks) · Hongwen (Olivia) Song · Sep 19, 2026

**Type:** Problem & solution

## Summary

Gray failures are partial customer-facing problems that can leave health checks green while a subset of users quietly fails. RADAR (Reliability Anomaly Detection, Alerting, and Root-cause analysis) addresses this gap by collecting error counts and distinct affected users, segmented by error code and region, then applying the unsupervised streaming SPOT model. SPOT learns normal behavior from the previous 14 days using a single risk parameter, while downstream stages enrich, filter, deduplicate, and route alerts before providing anomaly details and AI/BI Genie-backed dashboards for investigation. Running RADAR internally reduced incident-discovery time by 95% with over 90% precision and no human needed to spot the pattern. The post also describes a Databricks scaffold that maps metrics, detection, alerting, and root-cause analysis to platform components and can be assembled by an AI agent from a metric, scaffold, and prompt.

## Context

Gray failures affect only a subset of customers, so conventional health checks may remain normal. Customer reports can reveal the issue, but they are manual, delayed, and incomplete because many affected customers never file a ticket.

## Approach / What changed

RADAR records error counts and distinct affected users over time, broken down by error code and region. It applies the unsupervised streaming SPOT model, then enriches, filters, and deduplicates alerts before routing tickets to the appropriate engineering team. Root-cause analysis uses anomaly details and dashboards backed by AI/BI Genie. The Databricks implementation maps these stages to platform components and packages them in a Declarative Asset Bundle, with a scaffold that an AI agent can use alongside a metric and prompt.

## Takeaways

- RADAR detects unusual changes in error and affected-user time series without hand-tuned thresholds; SPOT learns normal behavior from 14 days of history and uses one risk parameter.
- The pipeline has four stages: reliability metrics, anomaly detection, alerting, and root-cause analysis. Alert enrichment, filtering, and deduplication help route actionable tickets without burying on-call responders in duplicates.
- Running RADAR internally produced a 95% reduction in incident-discovery time with over 90% precision and no human needed to identify the pattern.

## FAQ

### How does RADAR distinguish a gray failure from ordinary user errors?

It tracks error counts and the number of distinct users experiencing each error, segmented by error code and region. A sudden increase affecting many users can indicate a service problem even when the error appears to blame the user.

### What does SPOT contribute to the RADAR pipeline?

SPOT is an unsupervised, streaming anomaly-detection model used on each reliability-metric series. It learns normal behavior from the previous 14 days and requires a single risk parameter instead of manually maintained thresholds.

### What happens after RADAR detects an anomaly?

The alerting layer adds context, filters insignificant events, and deduplicates related alerts before filing a ticket with the responsible engineering team. The ticket includes anomaly deep-dive details and a link to a dashboard backed by AI/BI Genie.

**Tags:** [Anomaly Detection](https://yomu.fyi/topic/anomaly-detection), [Databricks](https://yomu.fyi/topic/databricks), [Delta Lake](https://yomu.fyi/topic/delta-lake), [MLflow](https://yomu.fyi/topic/mlflow), [Monitoring](https://yomu.fyi/topic/monitoring)

- Source: [Databricks](https://www.databricks.com/blog/radar-catch-gray-failures-anomaly-detection)
- Source URL: https://www.databricks.com/blog/radar-catch-gray-failures-anomaly-detection
- Ingested by Yomu: 2026-09-19T03:00:09.051Z

[Read original post](https://www.databricks.com/blog/radar-catch-gray-failures-anomaly-detection)