---
title: "Alerting Is a Pipeline, Not a Threshold"
description: "Alerting is an architecture, not a threshold: signals flow through detection, sharded evaluation, grouping, and routing. Here is how each layer decides whether a page is real."
---

# Alerting Is a Pipeline, Not a Threshold

**Type:** [Architecture](https://yomu.fyi/visual-explainers/category/architecture.md)

![Layered architecture diagram of an alerting pipeline: signal sources feed a detection layer that compares values to a baseline, then a sharded scheduler, a grouping layer, and routing to responders, with tuning knobs and a 24-hour preview panel on the side.](https://yomufyi.s3.us-east-1.amazonaws.com/visual-explainers/08f9a92a-a336-4a30-a6d9-7a2f3474280f/image-cccb580a-82b9-48d3-8685-03e6fd49d078.webp)

> Alerting is an architecture, not a threshold: signals flow through detection, sharded evaluation, grouping, and routing. Here is how each layer decides whether a page is real.

## Explanation

An alerting system is a pipeline with five bounded layers: signal sources, detection, scheduled evaluation, grouping, and routing. Detection compares a live value against a baseline rather than a fixed line, and the scheduler shards that evaluation across workers so tens of thousands of alerts can be checked on time, with backfill for skipped runs. Grouping turns many related signals into one actionable alert, and routing decides who is woken. Understanding the layers separately makes it clear why noise, missed pages, and slow detection are different problems with different fixes.

## FAQ

### Why evaluate against a baseline instead of a fixed threshold?

A fixed line breaks as traffic and metrics drift. ClickStack derives an expected value from a moving average of seven one-minute buckets plus a standard deviation from a two-hour baseline, with a 5% minimum change threshold so stable metrics do not page. Sensitivity, baseline duration, and direction of change stay configurable.

### How does an alerting system scale to tens of thousands of alerts?

Evaluation is scheduled work, so it shards. ClickStack distributes evaluation across multiple workers with an improved scheduler, adds a backfill mechanism for skipped evaluations, and exposes search, filtering, and evaluation histories in the alerts UI. Early-warning signals across serving tiers follow the same logic: warn before the hard limit, not at it.

### How do you cut alert noise without missing real incidents?

Group related signals before they page. Atlassian replaced one-to-one alerts with signal-based clustering on a standardized schema, and continuous accountability looks for repeated patterns of disregard rather than isolated violations. Previewing how often a rule would have fired over the previous 24 hours catches noisy rules before they reach on-call.

**Tags:** [Alerting](https://yomu.fyi/topic/alerting.md), [Anomaly Detection](https://yomu.fyi/topic/anomaly-detection.md), [Incident Response](https://yomu.fyi/topic/incident-response.md), [Metrics](https://yomu.fyi/topic/metrics.md), [Monitoring](https://yomu.fyi/topic/monitoring.md), [Observability](https://yomu.fyi/topic/observability.md)

## Related reading

### [Continuous accountability for Verified bots](https://yomu.fyi/post/continuous-accountability-for-verified-bots.md)
- Company: [Cloudflare](https://yomu.fyi/company/cloudflare.md)
- Author: Chris Tsoukaladelis
- Published: Oct 16, 2026
- Type: Announcement

Cloudflare introduces a system to continuously evaluate whether Verified bots honor website preferences, moving beyond identity verification to sustained trust by measuring compliance with robots.txt directives and HTTP 429 rate-limit responses over time. The system looks for repeated patterns of disregard rather than isolated violations, lowering a bot’s trust band if it ignores site preferences or fails to slow down after receiving rate-limit responses, while consistent compliance increases trust. Cloudflare previously de-listed Perplexity’s bot in 2025 for seeking to access opted-out resources and attempting to circumvent controls when blocked, illustrating the need for ongoing accountability. The company plans to share these trust scores publicly on Radar and eventually let customers choose the trust threshold for handling Verified bot traffic on their websites. This forms an accountability loop with Bot Preference Sync and Bot Management, ensuring trust is earned through sustained behavior rather than assumed from identity alone.


### [Reducing the noise: How Atlassian built an OCSF security lakehouse on Databricks](https://yomu.fyi/post/reducing-the-noise-how-atlassian-built-an-ocsf-security-lakehouse-on-d.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Taylor Kain, Zachary Schmerber (Atlassian), Niels Heijmans (Atlassian)
- Published: Oct 16, 2026
- Type: Problem & solution

Atlassian faced a security data pipeline ingesting 44 billion events daily from 20 years of growth and acquisitions, each with proprietary logging formats. To unify this fragmented telemetry, the team standardized on the Open Cybersecurity Schema Framework (OCSF) and built a security lakehouse on Databricks. They embedded the entire OCSF schema into Go structs (TypeGo), guaranteeing compile-time correctness and reducing AI hallucinations during automated transformation. Table optimization with partitioning and Z-ORDER cut query times by 95%, and Graviton clusters with Photon reduced cost and runtime by two-thirds. The result was a self-service platform where SOC analysts can onboard new data sources in days, and detection teams moved from one-to-one alerts to signal-based clustering using standardized OCSF values.


### [How our AI Research team uses AI Runtime](https://yomu.fyi/post/how-our-ai-research-team-uses-ai-runtime.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Ethan Tang, Ivan Zhou, Irene Dea, Steven Chen, Bruce Fontaine, Ben Hansen
- Published: Oct 16, 2026
- Type: Problem & solution

Databricks’ AI Research team uses AI Runtime (AIR) to manage hundreds of parallel research workstreams for products like Genie, Smart Routing, and AI Functions, training models from a few billion to nearly a trillion parameters. AIR performs preflight checks on GPU nodes to ensure health and communication before training, and monitors node health during execution to detect issues like NCCL failures or thermal throttling. The AIR CLI and custom Docker containers enable fast experimentation by launching jobs in seconds without rebuilding environments, supporting both supervised fine-tuning and reinforcement learning. MLflow integration automatically tracks experiments, while Unity Catalog FUSE mounting gives governed access to research data across workspaces without changing training code. The team has scaled to agent-led training, where AI agents propose changes, launch trials, and evaluate results, increasing training job volume by 8x across AI Functions research workloads.


### [2026 World Cup: Breaking Records, Spiky Charts and Reliability at Scale](https://yomu.fyi/post/2026-world-cup-breaking-records-spiky-charts-and-reliability-at-scale.md)
- Company: [Meta](https://yomu.fyi/company/meta.md)
- Author: Syamla Bandla
- Published: Oct 15, 2026
- Type: Problem & solution

The 2026 FIFA World Cup produced the largest planned stress test of Meta’s infrastructure, with WhatsApp setting an all-time record of over 30 million messages per second during the final and delivering 8.4 billion messages in its peak five-minute window. Traffic analysis revealed that surges in sharing primarily occurred during breaks and final whistles, not during active gameplay, shaping capacity decisions. Instagram and Facebook saw live video watch time spike 90% and 242% respectively, while recommendation traffic rose 80% week-over-week. To absorb these spikes, Meta layered pre-provisioned capacity buffers, country-level graceful degradation, cascade prevention, and early warning signals across all serving tiers. Notably, the team did not shift load between regions or freeze deployments, relying instead on year-round reliability enhancements and phased runbook execution.


### [Scaling ClickStack alerting with built-in anomaly detection](https://yomu.fyi/post/scaling-clickstack-alerting-with-built-in-anomaly-detection.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Published: Oct 15, 2026
- Type: Announcement

Managed ClickStack now supports statistical anomaly detection on query-builder charts using z-scores, allowing teams to move alerting workloads from tools like Grafana. The detection calculates the expected value from a moving average of seven one-minute buckets and the standard deviation from a two-hour baseline, with a 5% minimum change threshold to prevent false positives from stable metrics. Users configure sensitivity, baseline duration, and direction of change, and preview how often the alert would have fired over the previous 24 hours, with the chart updating as settings are adjusted. The alerting framework scales to tens of thousands of alerts through an improved scheduler and sharding that distributes evaluation across multiple workers, with a backfill mechanism for skipped evaluations and improved search, filtering, and evaluation histories in the alerts UI. Future plans include support for seasonality, median-based estimates for noisy data, and alerting on PromQL queries.