BCVK: A Local Test Bench for Bootable OS Images
BCVK is a local test bench for bootable container OS images. See how one OCI artifact moves through ephemeral, persistent, and disk-image modes before it reaches real hosts.
Loading visual explainer…
Alerting Is a Pipeline, Not a Threshold
Alerting is an architecture, not a threshold: signals flow through detection, sharded evaluation, grouping, and routing. Here is how each layer decides whether a page is real.
Explanation
An alerting system is a pipeline with five bounded layers: signal sources, detection, scheduled evaluation, grouping, and routing. Detection compares a live value against a baseline rather than a fixed line, and the scheduler shards that evaluation across workers so tens of thousands of alerts can be checked on time, with backfill for skipped runs. Grouping turns many related signals into one actionable alert, and routing decides who is woken. Understanding the layers separately makes it clear why noise, missed pages, and slow detection are different problems with different fixes.
FAQ
Why evaluate against a baseline instead of a fixed threshold?
A fixed line breaks as traffic and metrics drift. ClickStack derives an expected value from a moving average of seven one-minute buckets plus a standard deviation from a two-hour baseline, with a 5% minimum change threshold so stable metrics do not page. Sensitivity, baseline duration, and direction of change stay configurable.
How does an alerting system scale to tens of thousands of alerts?
Evaluation is scheduled work, so it shards. ClickStack distributes evaluation across multiple workers with an improved scheduler, adds a backfill mechanism for skipped evaluations, and exposes search, filtering, and evaluation histories in the alerts UI. Early-warning signals across serving tiers follow the same logic: warn before the hard limit, not at it.
How do you cut alert noise without missing real incidents?
Group related signals before they page. Atlassian replaced one-to-one alerts with signal-based clustering on a standardized schema, and continuous accountability looks for repeated patterns of disregard rather than isolated violations. Previewing how often a rule would have fired over the previous 24 hours catches noisy rules before they reach on-call.
The posts behind it
Related reading
Cloudflare · · AnnouncementContinuous accountability for Verified bots
Cloudflare introduces a system to continuously evaluate whether Verified bots honor website preferences, moving beyond identity verification to sustained trust by measuring compliance with robots.txt directives and HTTP 429 rate-limit responses over time. The system looks for repeated patterns of disregard rather than isolated violations, lowering a bot’s trust band if it ignores site preferences or fails to slow down after receiving rate-limit responses, while consistent compliance increases trust. Cloudflare previously de-listed Perplexity’s bot in 2025 for seeking to access opted-out resources and attempting to circumvent controls when blocked, illustrating the need for ongoing accountability. The company plans to share these trust scores publicly on Radar and eventually let customers choose the trust threshold for handling Verified bot traffic on their websites. This forms an accountability loop with Bot Preference Sync and Bot Management, ensuring trust is earned through sustained behavior rather than assumed from identity alone.
Databricks · · Problem & solutionReducing the noise: How Atlassian built an OCSF security lakehouse on Databricks
Atlassian faced a security data pipeline ingesting 44 billion events daily from 20 years of growth and acquisitions, each with proprietary logging formats. To unify this fragmented telemetry, the team standardized on the Open Cybersecurity Schema Framework (OCSF) and built a security lakehouse on Databricks. They embedded the entire OCSF schema into Go structs (TypeGo), guaranteeing compile-time correctness and reducing AI hallucinations during automated transformation. Table optimization with partitioning and Z-ORDER cut query times by 95%, and Graviton clusters with Photon reduced cost and runtime by two-thirds. The result was a self-service platform where SOC analysts can onboard new data sources in days, and detection teams moved from one-to-one alerts to signal-based clustering using standardized OCSF values.
- Databricks · · Problem & solution
How our AI Research team uses AI Runtime
Databricks’ AI Research team uses AI Runtime (AIR) to manage hundreds of parallel research workstreams for products like Genie, Smart Routing, and AI Functions, training models from a few billion to nearly a trillion parameters. AIR performs preflight checks on GPU nodes to ensure health and communication before training, and monitors node health during execution to detect issues like NCCL failures or thermal throttling. The AIR CLI and custom Docker containers enable fast experimentation by launching jobs in seconds without rebuilding environments, supporting both supervised fine-tuning and reinforcement learning. MLflow integration automatically tracks experiments, while Unity Catalog FUSE mounting gives governed access to research data across workspaces without changing training code. The team has scaled to agent-led training, where AI agents propose changes, launch trials, and evaluate results, increasing training job volume by 8x across AI Functions research workloads.
Meta · · Problem & solution2026 World Cup: Breaking Records, Spiky Charts and Reliability at Scale
The 2026 FIFA World Cup produced the largest planned stress test of Meta’s infrastructure, with WhatsApp setting an all-time record of over 30 million messages per second during the final and delivering 8.4 billion messages in its peak five-minute window. Traffic analysis revealed that surges in sharing primarily occurred during breaks and final whistles, not during active gameplay, shaping capacity decisions. Instagram and Facebook saw live video watch time spike 90% and 242% respectively, while recommendation traffic rose 80% week-over-week. To absorb these spikes, Meta layered pre-provisioned capacity buffers, country-level graceful degradation, cascade prevention, and early warning signals across all serving tiers. Notably, the team did not shift load between regions or freeze deployments, relying instead on year-round reliability enhancements and phased runbook execution.
Clickhouse · · AnnouncementScaling ClickStack alerting with built-in anomaly detection
Managed ClickStack now supports statistical anomaly detection on query-builder charts using z-scores, allowing teams to move alerting workloads from tools like Grafana. The detection calculates the expected value from a moving average of seven one-minute buckets and the standard deviation from a two-hour baseline, with a 5% minimum change threshold to prevent false positives from stable metrics. Users configure sensitivity, baseline duration, and direction of change, and preview how often the alert would have fired over the previous 24 hours, with the chart updating as settings are adjusted. The alerting framework scales to tens of thousands of alerts through an improved scheduler and sharding that distributes evaluation across multiple workers, with a backfill mechanism for skipped evaluations and improved search, filtering, and evaluation histories in the alerts UI. Future plans include support for seasonality, median-based estimates for noisy data, and alerting on PromQL queries.
More like this
More architecture explainers
B2B Integration Architecture: Four Layers Between Your Service and the Customer's Stack
In B2B, most of the engineering lives in the seams: the tenancy boundary, the integration surface, state sync, and the cross-cutting trust rail. This diagram maps those four layers end to end.
AI-DLC Architecture: One Spec Spine Through Inception, Design, and Construction
AI-DLC is an AI-driven development lifecycle that runs delivery through Inception, Design, and Construction, with specs, steering files, and hooks keeping agents aligned to agreed intent.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
The best engineering reads, once a week
One useful email. Unsubscribe anytime.