---
title: "Observability"
description: "250 posts about Observability, summarised, each linking to the original."
---

# Observability
> 250 posts about Observability, summarised, each linking to the original.

## Articles

### [How Databricks Uses AI to Accelerate Incident Investigation](https://yomu.fyi/post/how-databricks-uses-ai-to-accelerate-incident-investigation.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Avijeet Gupta, Bhuban Seth, Kusum Madarasu
- Published: Aug 24, 2026
- Type: Explainer

Databricks describes AI SRE, an AI-powered debugging agent for operating hundreds of microservices across more than 1,500 Kubernetes clusters, 70+ regions, and three clouds. It addresses incident investigations by assembling context from platform health, service telemetry, changes, dependencies, and team-specific runbooks, with automatic triage followed by interactive, natural-language investigation. The platform uses layered primitives, purpose-built APIs, and a core orchestration engine; team-owned agentic runbooks encode domain procedures, while controlled access and evidence links support auditing. The post reports support for more than 150 teams, 250+ weekly active users, and over 2,000 daily investigations, with users saving several hours of debugging time.


### [NVIDIA BlueField-4 Powers New Scale-In Network Infrastructure for Agentic AI Factories](https://yomu.fyi/post/nvidia-bluefield-4-powers-new-scale-in-network-infrastructure-for-agen.md)
- Company: [NVIDIA](https://yomu.fyi/company/nvidia-developer-blog.md)
- Author: Michelle Horton
- Published: Aug 24, 2026
- Type: Explainer

Agentic AI factories connect diverse users, agents, applications, data sources, and storage systems to massively accelerated compute, making dedicated DPU processing essential for line-rate networking, storage, and security. Scale-In evolves north-south access into a coordinated infrastructure domain built with BlueField-4, NVIDIA DOCA, and Spectrum-X Ethernet. BlueField-4 combines programmable control-plane processing with inline data-path engines, a 64-core Grace CPU, LPDDR5X memory, PCIe Gen6 host connectivity, and an 800 Gb/s network interface. The architecture offloads security, tenant isolation, storage protocols, provisioning, and telemetry from host CPUs; the source states that Spectrum-X storage paths can provide up to 1.45x the throughput of off-the-shelf Ethernet, while independent control and observability support more consistent operations as AI infrastructure scales.


### [Ensuring reliable OpenTelemetry ingestion at scale](https://yomu.fyi/post/ensuring-reliable-opentelemetry-ingestion-at-scale.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Published: Aug 24, 2026
- Type: Problem & solution

LogHouse’s OpenTelemetry pipeline ingests 50 million events per second and stores 177 PiB of uncompressed data, but its initial agent-to-gateway design could not reliably absorb ClickHouse backpressure. In-memory queues caused instability during short-lived spikes or database outages, while local collector write-ahead logs made gateways stateful, slowed recovery, delayed fresh telemetry behind FIFO backlogs, and took about four hours to drain 1 TiB. The team rejected adding Kafka because operating a new tier-zero service across cloud regions was not justified for this use case, then designed a custom architecture using blob storage, the OTel failover connector, overflow notifications, and a catchup collector. In staging, failover redirected 120,000 events per second to S3 without changing gateway resource usage, and recovery resumed ClickHouse writes while the catchup collector drained the queue; the design scaled from 10 million to 50 million events per second, with added regional setup and less precise recovery visibility.


### [What else runs on your Postgres server, and how do we stop it from taking the database down?](https://yomu.fyi/post/what-else-runs-on-your-postgres-server-and-how-do-we-stop-it-from-taki.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Published: Aug 21, 2026
- Type: Problem & solution

The Postgres VM described here runs PgBouncer instances, a backup agent, metrics exporters, a local Prometheus, a log collector, and watchdog timers alongside the database. Those processes share machine-wide resources outside Postgres’s huge-page-backed shared\_buffers, so leaks, CPU saturation, or disk exhaustion can threaten database availability. ClickHouse Managed Postgres places supporting services in a cgroup v2 slice and combines Go runtime GOMEMLIMIT targets with systemd memory.high and memory.max enforcement. GOMEMLIMIT encourages earlier garbage collection, memory.high applies reclaim and throttling, and memory.max provides a hard ceiling whose OOM scope excludes Postgres because it uses a separate cgroup. CPU weights, bounded buffers, log and metric controls, disk thresholds, and exemptions for replication and monitoring sessions extend isolation across other failure modes; these controls are enabled by default.


### [Improved failure reports on Red Hat OpenShift with the event-driven diagnostic operator](https://yomu.fyi/post/improved-failure-reports-on-red-hat-openshift-with-the-event-driven-di.md)
- Company: [Red Hat](https://yomu.fyi/company/red-hat.md)
- Author: Periyamaruthu Mohanraj
- Published: Aug 21, 2026
- Type: Problem & solution

During major infrastructure incidents, critical system logs and diagnostic evidence are frequently overwritten, lost, or erased by emergency recovery efforts before engineers can investigate. To solve this visibility gap in multi-cluster environments, an open-source event-driven diagnostic operator was developed as an automatic first information report collector. Deployed on a central hub cluster, the operator monitors Kubernetes warning events rather than relying on heavy polling. When an event matches a predefined regular expression pattern, the operator identifies the target spoke cluster, copies its kubeconfig secret, and spins up an independent must-gather job. Diagnostic data is written to off-site or shared storage with a configurable retention window, preserving forensic context without human intervention.


### [How AgentFlo built AI sales agents with Amazon Bedrock AgentCore – Part 2](https://yomu.fyi/post/how-agentflo-built-ai-sales-agents-with-amazon-bedrock-agentcore-part-2.md)
- Company: [AWS](https://yomu.fyi/company/aws.md)
- Author: Muhammad Musab Iqbal
- Published: Aug 21, 2026
- Type: Problem & solution

AgentFlo designed an architecture using Amazon Bedrock AgentCore and AWS serverless infrastructure to address traffic spikes and build trust in autonomous commerce agents. Security operates across three stages: AWS Fargate filters prompt injections and opt-outs, AgentCore Gateway enforces deterministic Cedar authorization policies during tool execution, and post-turn privacy filters screen outputs. Session isolation is maintained across merchants through AgentCore runtime microVMs, while AgentCore Observability and Amazon CloudWatch provide end-to-end tracing. Data reliability and long conversation journeys rely on Amazon DynamoDB context storage, Amazon Bedrock Knowledge Bases, and Amazon S3 Vector embeddings. Early deployment data showed a 12 percent net revenue uplift.


### [August 2026 newsletter](https://yomu.fyi/post/august-2026-newsletter.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Published: Aug 21, 2026
- Type: Announcement

The August 2026 newsletter rounds up ClickHouse product releases, research, managed services, observability, community work, and upcoming events. It introduces Andy Pavlo’s arrival to establish ClickHouse Labs, describes the 26.7 release’s join optimizations, token-position text indexes, and EXPLAIN ANALYZE, and reports a 40× phrase-search improvement on a Hacker News dataset. It also covers an interactive ClickBench playground for more than 100 database systems, running isolated workloads in Firecracker microVMs on AWS EC2, plus guidance on combining Postgres for mutable data with ClickHouse for immutable analytical workloads. Additional items address Managed Postgres notifications and backups, TTL-related disk growth, vector-search trade-offs, and hdx-evals, which found a purpose-built ClickStack MCP server scored 7–20 percentage points higher than direct SQL access in five reproducible scenarios.


### [How Sony LIV uses ClickHouse Cloud to deliver live streaming analytics at billion-row scale](https://yomu.fyi/post/how-sony-liv-uses-clickhouse-cloud-to-deliver-live-streaming-analytics.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Published: Aug 18, 2026
- Type: Problem & solution

Sony LIV’s legacy analytics environment combined batch pipelines, Elasticsearch, BigQuery, and other telemetry stores, creating slow, costly, fragmented analysis as streaming activity reached billions of events per day. During live sports, the platform needed sub-second queries, high ingestion throughput, cost-efficient long-term storage, and horizontal scaling for sudden traffic spikes. Sony LIV consolidated these workloads on ClickHouse Cloud: client, streaming, CDN, and advertising systems send events through Amazon Kinesis and ClickPipes into a ClickHouse cluster serving dashboards, product analytics, QoE, and SRE workflows. The source reports that queries previously taking tens of seconds to minutes now run in sub-second to low-single-digit seconds, improving operational responsiveness, engineering productivity, and the handling of major live-event traffic.


### [When it comes to Governance, Retailers need a control plane for context](https://yomu.fyi/post/when-it-comes-to-governance-retailers-need-a-control-plane-for-context.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: BJ Sullivan, Sarah Duffy, Tayo Olabumuyi, Logan Boyd
- Published: Aug 18, 2026
- Type: Explainer

Retailers are moving from isolated generative-AI pilots to enterprise-wide adoption, where each store, headquarters, and digital workflow needs different data, permissions, models, and cost controls. The post argues that useful AI depends on governed business context, while fragmented tools can produce separate provider contracts, logs, permission models, cost centers, and answers to the same question. It proposes an open control plane that centralizes governance and model access without forcing employees into one application or vendor. The described Databricks design combines Unity Catalog for data and business context, Unity AI Gateway for routing and oversight, Foundation Model APIs for commercial, open-source, and custom models, and Genie for natural-language queries over governed data; the intended result is faster experimentation and deployment with visibility into access, spending, lineage, and auditability.


### [What's New with Monitoring in PostgreSQL 19](https://yomu.fyi/post/what-s-new-with-monitoring-in-postgresql-19.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Published: Aug 18, 2026
- Type: Explainer

PostgreSQL 19, still in beta when this was written, adds monitoring and observability changes spanning lock contention, logging, WAL, wait events, remote messages, and multixacts. The default for log\_lock\_waits changes to on, while log\_min\_messages gains per-process-type levels and autoanalyze receives its own duration setting. A new wal\_fpi\_bytes counter exposes full-page-image volume cluster-wide, per session, per query, and in VACUUM and ANALYZE logs; new wait events expose WAL write, flush, and replay stages plus COPY file, pipe, and program I/O. Remote server messages now use normal PostgreSQL logging with origin prefixes, and pg\_get\_multixact\_stats() surfaces multixact usage that previously required filesystem inspection and educated guessing. The post also lists upgrade work for parsers, metric collectors, wait-event dictionaries, configuration UIs, and alert thresholds, while noting that final release notes may differ.


### [Connect client traces to your logs](https://yomu.fyi/post/connect-client-traces-to-your-logs.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: katerina\_skroumpelou, steven\_eubank
- Published: Aug 18, 2026
- Type: Announcement

supabase-js now supports W3C Trace Context propagation so a client trace can share its trace\_id with Supabase API Gateway and Edge Function logs. Previously, browser and server records had no shared identifier, making cross-boundary debugging depend on manually aligning timestamps. When tracePropagation is enabled, the SDK attaches traceparent, tracestate, and baggage to requests for Supabase domains; Supabase preserves the trace\_id in generated logs, and W3C-compliant tracers such as OpenTelemetry, Sentry, Datadog, Honeycomb, or Grafana can continue the trace server-side. The setup requires a separately configured tracer, the tracing entry-point import, an active span, and a registered TracerProvider; propagation is sampling-aware by default, adds no OpenTelemetry code to the main bundle, and is available in supabase-js 2.112.0 or later, with CDN/UMD builds excluded.


### [How Cloudflare detects MCP traffic and helps secure it](https://yomu.fyi/post/how-cloudflare-detects-mcp-traffic-and-helps-secure-it.md)
- Company: [Cloudflare](https://yomu.fyi/company/cloudflare.md)
- Author: AJ Gerstenhaber
- Published: Aug 14, 2026
- Type: Announcement

Traditional security boundaries assume human speed and judgment, but AI agents execute nondeterministic actions rapidly and can invoke unapproved Model Context Protocol servers through standard HTTPS traffic. To mitigate this shadow traffic, Cloudflare One introduces detection and inspection capabilities across clients, network boundaries, and MCP servers. Cloudflare Gateway leverages protocol-level headers and JSON-RPC inspection to identify direct MCP connections, verify managed paths, and enforce access through approved MCP Server Portals. Furthermore, Cloudflare Agents SDK v0.20.0 implements the stateless 2026-07-28 MCP specification while providing fallback handshakes for legacy servers. Organizations can combine Gateway traffic source rules with server middleware like WriteGuard to block unauthorized tool invocations before execution.


### [Close the Database Blind Spot with MongoDB Atlas Observability](https://yomu.fyi/post/close-the-database-blind-spot-with-mongodb-atlas-observability.md)
- Company: [MongoDB](https://yomu.fyi/company/mongodb.md)
- Published: Aug 14, 2026
- Type: Announcement

MongoDB Atlas introduces OpenTelemetry Metrics Sink, a push-based stream that delivers high-fidelity Atlas metrics to OpenTelemetry-compatible platforms, enabling database telemetry to flow alongside application and infrastructure metrics. This closes the gap where database telemetry was often disconnected from application and infrastructure monitoring, forcing operators to jump between dashboards. The push-based delivery eliminates the complexity of pull-based collection, custom exporters, and polling infrastructure, reducing operational overhead and avoiding API rate limits. It supports configuration via the Atlas UI or Terraform, works with custom OTel endpoints, Dynatrace, and New Relic, and is available for dedicated M10+ clusters; a test metric is emitted on creation or update to verify it is working. Combined with Atlas Log Integration and upcoming Events Integration, teams gain unified visibility into database health alongside other signals, accelerating incident response and root cause analysis.


### [Total eclipse of the Internet: traffic impacts in Iceland, Spain, and Portugal](https://yomu.fyi/post/total-eclipse-of-the-internet-traffic-impacts-in-iceland-spain-and-por.md)
- Company: [Cloudflare](https://yomu.fyi/company/cloudflare.md)
- Author: Sabina Zejnilovic
- Published: Aug 13, 2026
- Type: Explainer

On August 12, a total solar eclipse crossed the North Atlantic and Europe, passing directly over Iceland, northern Spain, and Portugal. Cloudflare Radar analyzed HTTP request volumes in five-minute intervals across affected countries, comparing the data against a baseline derived from the previous three Wednesdays. Traffic reductions aligned closely with peak solar obscuration, with totality regions recording drops between 15% and 30% and specific areas shifting by up to -46.7%. Once the sun reappeared, request volumes rebounded to normal levels within minutes as users resumed device activity. The observed traffic declines were driven by aggregate human behavior rather than network failures.


### [Certificate Transparency Monitoring is now generally available](https://yomu.fyi/post/certificate-transparency-monitoring-is-now-generally-available.md)
- Company: [Cloudflare](https://yomu.fyi/company/cloudflare.md)
- Author: Jenny Yang
- Published: Aug 13, 2026
- Type: Problem & solution

Cloudflare announced the general availability of Certificate Transparency Monitoring alongside a filtering mechanism to eliminate noisy alert emails. Previously, the monitoring system alerted domain owners for all Certificate Transparency log entries, including routine, automated renewals for Universal SSL, Advanced Certificate Manager, and backup certificates. Because the alerting flow and certificate ordering service operated independently, existing identifiers like TBSCertificate hashes arrived too late to verify pre-certificates. To resolve this race condition, the ordering service now computes an SHA-256 hash of the DER-encoded SubjectPublicKeyInfo structure at key generation. When the alerting service discovers a log entry, it recomputes this hash to verify ownership against the ordering database, suppressing alerts for Cloudflare-managed certificates while preserving alerts for external or unexpected issuances.


### [ClickStack and Hud bring runtime intelligence to AI-powered development](https://yomu.fyi/post/clickstack-and-hud-bring-runtime-intelligence-to-ai-powered-developmen.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Published: Aug 13, 2026
- Type: Announcement

ClickHouse and Hud announce an integration that combines ClickStack’s service-level observability with Hud’s function-level runtime forensics for AI-assisted software development. As coding agents generate more code, the post identifies the central challenge as assessing production risk, detecting regressions, and giving engineers enough context to fix them. Teams connect the ClickStack and Hud MCP servers to a single coding agent, while shared trace IDs correlate OpenTelemetry traces and logs with Hud’s function-level metrics and forensic data; Hud’s Runtime Code Sensor requires no manual instrumentation and runs with low overhead in production. The resulting workflows support pre-deployment risk assessment against production behavior, release monitoring, auto-triage and auto-fix, and rollback or code-level remediation when issues arise.


### [Automated agent triage with Agent Tracing and Claude Routines](https://yomu.fyi/post/automated-agent-triage-with-agent-tracing-and-claude-routines.md)
- Company: [Sentry](https://yomu.fyi/company/sentry.md)
- Author: Trevor Elkins
- Published: Aug 13, 2026
- Type: Problem & solution

Evaluating Sentry's AI agent Seer previously required manually querying evaluation data and parsing raw JSON traces to find tool failures, latency spikes, and hallucinations. To automate this triage, the team deployed a daily Claude Routine connected to Sentry using a Model Context Protocol server. Each morning, the routine analyzes hundreds of overnight agent conversations, reviews aggregate error metrics, samples failed and successful conversation spans, and inspects the agent codebase for context. Claude evaluates whether final decisions match underlying reasoning and creates Linear tickets for newly discovered issues. This daily automated analysis accelerated the feedback loop, allowing engineers to identify efficiency issues like repeated malformed repository queries and deploy same-day fixes.


### [Adobe Firefly: Simplified observability with Amazon Managed Prometheus](https://yomu.fyi/post/adobe-firefly-simplified-observability-with-amazon-managed-prometheus.md)
- Company: [AWS](https://yomu.fyi/company/aws.md)
- Author: Dev Arora
- Published: Aug 13, 2026
- Type: Problem & solution

Adobe Firefly powers generative creative features using a large-scale GPU training infrastructure hosted on Amazon Elastic Kubernetes Service. As workloads scaled to thousands of nodes and GPUs, Adobe's self-managed Prometheus setup struggled with query latency across high-cardinality telemetry data. To resolve these performance bottlenecks, the team moved two million critical time series metrics to Amazon Managed Service for Prometheus while using managed scrapers alongside their existing infrastructure. This setup accelerated query response times by more than 28x over 24-hour windows, reducing query execution times to approximately ten seconds. Additionally, the migration expanded practical monitoring windows from six hours to 24 hours, giving teams reliable visibility across long-running distributed training jobs.


### [How to Choose Full-Stack Observability for NVIDIA AI Factories](https://yomu.fyi/post/how-to-choose-full-stack-observability-for-nvidia-ai-factories.md)
- Company: [NVIDIA](https://yomu.fyi/company/nvidia-developer-blog.md)
- Author: Jorge Cardoso
- Published: Aug 12, 2026
- Type: Explainer

AI infrastructure spans compute, networking, storage, orchestration, and applications, so a symptom at one layer may originate elsewhere. The post proposes selecting observability tools by failure domain, mapping components to telemetry sources, and reducing signals to a top-k alert set in a unified Prometheus/Grafana triage dashboard. For a DGX cluster using InfiniBand, BCM, and Slurm, it recommends Redfish/IPMI, DCGM, NVSM, UFM, and BCM, while deferring NetQ, NMX, Run:ai, and NIM until their corresponding capabilities are needed. The framework emphasizes DCGM for GPU metrics, NVSM for DGX system health, UFM for InfiniBand, and BCM for cluster and job aggregation, with specialized tools retaining low-level telemetry. An elevated InfiniBand bit error rate illustrates how retransmissions can slow one rank during NCCL all-reduce, causing synchronized training throughput to follow the slowest rank; alerts should connect concise signals to owners and remediation actions.


### [How iFood built its agentic security platform on ClickHouse Cloud](https://yomu.fyi/post/how-ifood-built-its-agentic-security-platform-on-clickhouse-cloud.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Published: Aug 12, 2026
- Type: Problem & solution

iFood’s security incident response team needed high-volume log ingestion, long-term retention, and rapid access to months of history, but its Databricks-based platform became costly to scale beyond 130 TB and imposed query timeouts. The team ran a production-data proof of concept and rebuilt its tooling around ClickHouse Cloud on AWS: logs land in S3 and flow through ClickPipes into ClickHouse, with Querybook preserving notebook-based investigations and ClickStack planned for operational metrics. With ClickHouse Cloud, iFood reports 9–16x faster queries, expected cost reductions of 40–50%, and data freshness improving from hourly batches to 2–10 minutes; alerts that previously ran hourly can now run about every 10 minutes. That foundation also supports agentic threat hunting: multiple sub-agents query more than 30 TB in parallel, reducing a hunt from about a week to roughly two hours, while Langfuse is used to inspect and tune agent analyses.


[Newer posts](https://yomu.fyi/topic/observability/page/2.md) · [Older posts](https://yomu.fyi/topic/observability/page/4.md)