The three villains to agentic observability: retention, sampling and rollups

Summary

Observability practices increasingly depend on data sampling, short retention windows, and rollups to control storage costs. While human operators historically compensated for these compromises with intuition, AI agents lack implicit operational context and require full-fidelity telemetry to diagnose issues and justify conclusions. Head sampling drops unobserved spans early, tail sampling risks missing implicit non-error anomalies, and aggressive log expiration prevents long-term cyclical trend analysis. Addressing these constraints requires decoupling storage and compute via high-compression object storage and dedicated compute pools. Deploying real-time analytical databases such as ClickHouse enables high-concurrency, low-latency SQL querying, log clustering, and structured in-database summarization necessary for automated reasoning loops.

Context

Storage engine scaling limitations and high-cardinality costs force teams to limit log retention, sample traces, and aggregate metrics, depriving AI diagnostic agents of critical context.

Approach / What changed

Store raw observability telemetry using high-ratio compression on object storage, provision workload-specific compute pools, and leverage high-concurrency analytical SQL engines like ClickHouse for in-database summarization.

Takeaways

• Combining object storage at roughly $0.025 per gigabyte with 50x compression makes 30-day, 60-day, or annual raw data retention economically feasible.
• Both head sampling and tail sampling drop telemetry data at ingestion, distorting metrics aggregates and breaking wide-event analysis required for agent reasoning.
• Agents need high query concurrency, sub-second latency, and standard SQL access to execute iterative root cause analysis loops without datastore bottlenecks.

Related reading