Using observability data to prevent incidents

Summary

Engineering teams often remain reactive not because they lack telemetry, but because metrics, logs, traces, error budgets, and SLO burn rates are difficult to query at decision speed. The post frames reliability intelligence as trend analysis that identifies accumulating risk before a user-facing incident, contrasting it with observability focused on current system state. It proposes unified telemetry access and natural-language self-service, using Databricks Genie to examine p99 latency, error-budget burn, deployment frequency, incident rates, dependencies, and capacity across services. The Trade Desk’s Genie Router handles questions across more than 10,000 tables, while the post says ad-hoc analysis runs five times faster with Genie and can support mitigation before a page fires.

Context

Engineering organizations have observability data and instrumentation but lack on-demand access to query reliability trends across services at the pace of engineering decisions. This delay keeps teams focused on reactive incident response, while accumulating risk, incident frequency, business costs, and lost R&D capacity remain insufficiently addressed.

Approach / What changed

The proposed approach combines centralized telemetry with self-service, natural-language querying through Databricks Genie. Leaders can interrogate metrics, logs, traces, deployment records, incident history, service dependencies, DORA metrics, capacity, and cost data to identify trends and prioritize mitigation before an incident. The Trade Desk’s Genie Router directs questions across more than 10,000 tables.

Takeaways

• Reliability intelligence uses telemetry trend analysis to identify likely reliability risk before it becomes a user-facing incident, whereas observability is described as showing what is happening now.
• Databricks Genie supports natural-language questions about p99 latency increases, error-budget burn rates, deployment frequency, incident rates, service dependencies, and capacity across engineering data.
• The Trade Desk built a Genie Router for an environment with more than 10,000 tables, and the post states that ad-hoc analysis runs five times faster with Genie.

Related reading