# Observability > 32 posts about Observability, summarised, each linking to the original. ## Articles ### [OpenTelemetry Comes to IntelliJ IDEA, GoLand, PyCharm, and WebStorm](https://yomu.fyi/post/opentelemetry-comes-to-intellij-idea-goland-pycharm-and-webstorm.md) - Company: [Jetbrains](https://yomu.fyi/company/jetbrains.md) - Author: Egor Klimov - Published: Aug 26, 2026 With the 2026.2 release, JetBrains expanded its OpenTelemetry plugin from Rider to IntelliJ IDEA, GoLand, PyCharm, and WebStorm. The tool captures logs, metrics, traces, and service maps locally without requiring an external observability backend. Developers can search structured log records, plot metric values over time, inspect distributed spans, and verify communication paths across databases and message queues during local runs. To ingest telemetry, the plugin automatically configures OpenTelemetry Protocol environment variables for supported run configurations and terminal sessions, or accepts forwarded data from existing collectors. Additionally, experimental Model Context Protocol support allows AI coding agents to query gathered logs, spans, and service topology. ### [How Cloudflare detects MCP traffic and helps secure it](https://yomu.fyi/post/how-cloudflare-detects-mcp-traffic-and-helps-secure-it.md) - Company: [Cloudflare](https://yomu.fyi/company/cloudflare.md) - Author: AJ Gerstenhaber - Published: Aug 14, 2026 Traditional security boundaries assume human speed and judgment, but AI agents execute nondeterministic actions rapidly and can invoke unapproved Model Context Protocol servers through standard HTTPS traffic. To mitigate this shadow traffic, Cloudflare One introduces detection and inspection capabilities across clients, network boundaries, and MCP servers. Cloudflare Gateway leverages protocol-level headers and JSON-RPC inspection to identify direct MCP connections, verify managed paths, and enforce access through approved MCP Server Portals. Furthermore, Cloudflare Agents SDK v0.20.0 implements the stateless 2026-07-28 MCP specification while providing fallback handshakes for legacy servers. Organizations can combine Gateway traffic source rules with server middleware like WriteGuard to block unauthorized tool invocations before execution. ### [Total eclipse of the Internet: traffic impacts in Iceland, Spain, and Portugal](https://yomu.fyi/post/total-eclipse-of-the-internet-traffic-impacts-in-iceland-spain-and-por.md) - Company: [Cloudflare](https://yomu.fyi/company/cloudflare.md) - Author: Sabina Zejnilovic - Published: Aug 13, 2026 On August 12, a total solar eclipse crossed the North Atlantic and Europe, passing directly over Iceland, northern Spain, and Portugal. Cloudflare Radar analyzed HTTP request volumes in five-minute intervals across affected countries, comparing the data against a baseline derived from the previous three Wednesdays. Traffic reductions aligned closely with peak solar obscuration, with totality regions recording drops between 15% and 30% and specific areas shifting by up to -46.7%. Once the sun reappeared, request volumes rebounded to normal levels within minutes as users resumed device activity. The observed traffic declines were driven by aggregate human behavior rather than network failures. ### [Certificate Transparency Monitoring is now generally available](https://yomu.fyi/post/certificate-transparency-monitoring-is-now-generally-available.md) - Company: [Cloudflare](https://yomu.fyi/company/cloudflare.md) - Author: Jenny Yang - Published: Aug 13, 2026 Cloudflare announced the general availability of Certificate Transparency Monitoring alongside a filtering mechanism to eliminate noisy alert emails. Previously, the monitoring system alerted domain owners for all Certificate Transparency log entries, including routine, automated renewals for Universal SSL, Advanced Certificate Manager, and backup certificates. Because the alerting flow and certificate ordering service operated independently, existing identifiers like TBSCertificate hashes arrived too late to verify pre-certificates. To resolve this race condition, the ordering service now computes an SHA-256 hash of the DER-encoded SubjectPublicKeyInfo structure at key generation. When the alerting service discovers a log entry, it recomputes this hash to verify ownership against the ordering database, suppressing alerts for Cloudflare-managed certificates while preserving alerts for external or unexpected issuances. ### [How Standardizing Product Telemetry Reduced Time to Insight by 97%](https://yomu.fyi/post/how-standardizing-product-telemetry-reduced-time-to-insight-by-97.md) - Company: [Salesforce](https://yomu.fyi/company/salesforce.md) - Author: Scott Nyberg - Published: Aug 11, 2026 Salesforce faced scaling bottlenecks when individual engineering teams maintained fragmented, custom telemetry pipelines requiring manual data transformations and dashboard creation every release. To resolve these silos, the engineering organization established the Product Data Platform (PDP) using a standardized custom schema built atop Monitoring Cloud infrastructure. This framework mandates core telemetry fields for consistent metric generation while offering governed optional attributes for product-specific flexibility. An AI-driven Model Context Protocol tool assists developers by recommending compliant instrumentation patterns directly within their workflows. Today, the unified platform processes 45 billion rows of data daily across 19,000 distinct events and 2,000 product features, reducing dashboard refresh latency by 97 percent from one month to daily updates. ### [Everything we launched during Agents Week](https://yomu.fyi/post/everything-we-launched-during-agents-week.md) - Company: [Cloudflare](https://yomu.fyi/company/cloudflare.md) - Author: Shelley Jones - Published: Aug 10, 2026 Agents Week introduced a series of products and primitives across Cloudflare to support an Agentic Internet where autonomous software and humans interact. Developing capable agents requires addressing technical challenges spanning identity, communication, orchestration, memory, observability, and security. Cloudflare introduced the Agent Development Lifecycle alongside tools like Cloudflare Agents for live tracing, replay, and human-in-the-loop approvals. The releases also included programmable CI/CD pipelines capable of repairing failures, Cloudflare Wallets for agent transactions, and Kitesurf, an agent-first browser running in V8 isolates. New protocols and tools like WebMCP, MCPv2, and Cloudflare AI Search aim to make web data and applications readable, discoverable, and searchable by autonomous software. ### [Unifying Workers AI and AI Gateway into a single AI control plane](https://yomu.fyi/post/unifying-workers-ai-and-ai-gateway-into-a-single-ai-control-plane.md) - Company: [Cloudflare](https://yomu.fyi/company/cloudflare.md) - Author: Michelle Chen - Published: Aug 7, 2026 Workers AI and AI Gateway are converging into a unified AI control plane to simplify model access, billing, and observability. Developers can now access both services through a single entrypoint using unified Workers bindings and the /ai/ REST endpoint. Specifying a default gateway automatically instruments requests with payload logging, token tracking, and cost attribution without manual dashboard setup. Cloudflare has also rolled out unified prepaid billing, allowing teams to spend AI Gateway credits on Workers AI models while unlocking higher rate limits. Planned future features include model-first routing for automated provider failover and smart routing powered by task classifiers on Workers AI. ### [From ranking to recommended: get your site ready to thrive in the age of AI agents](https://yomu.fyi/post/from-ranking-to-recommended-get-your-site-ready-to-thrive-in-the-age-o.md) - Company: [Cloudflare](https://yomu.fyi/company/cloudflare.md) - Author: Matthew Conroy - Published: Aug 6, 2026 Discoverability is shifting from traditional search engine rankings to recommendations generated by AI agents acting on behalf of users. To address this shift, Cloudflare integrated Agent Readiness diagnostics and an Answer Engine Optimization tool into its dashboard. The Diagnostics feature evaluates hostnames against machine-readable standards across tiers ranging from robots.txt and Markdown rendering to OAuth discovery and agent payment protocols. Simultaneously, the Answer Engine Optimization tool measures how frequently leading assistants like Claude and GPT cite, mention, and prominently position brands across industry categories. By evaluating prompt responses using Cloudflare AI Gateway and Workers AI alongside access logs, site owners can systematically track agent traffic, benchmark against competitors, and remediate crawl blockers. ### [How Salesforce Eliminated Single-Region Risk and Reduced Downtime Blast Radius at 4B Metrics/Min](https://yomu.fyi/post/how-salesforce-eliminated-single-region-risk-and-reduced-downtime-blas.md) - Company: [Salesforce](https://yomu.fyi/company/salesforce.md) - Author: Scott Nyberg - Published: Aug 5, 2026 Salesforce redesigned its internal observability platform, Argus, to eliminate single-region failure risks while ingesting approximately 4 billion metrics per minute. Operating out of a single AWS region previously created a global blast radius and generated high cross-region data transfer costs. To mitigate these risks without the prohibitive expense of full multi-region replication, the engineering team implemented a geo-local architecture that processes and stores telemetry closer to its origin across production geographies. A newly introduced federation query layer relies on Elasticsearch mappings to route queries selectively to relevant regional clusters, avoiding broad fan-out. The platform also adopted metadata caching for wildcard queries and implemented HTTP 206 partial-response handling to maintain visibility when individual regions experience downtime. ### [Eval-driven development: Lessons from evaluating GenAI at scale](https://yomu.fyi/post/eval-driven-development-lessons-from-evaluating-genai-at-scale.md) - Company: [Airbnb](https://yomu.fyi/company/airbnb.md) - Author: Rohit Girme - Published: Jul 28, 2026 Airbnb outlines its framework for eval-driven development (EDD), treating GenAI evaluation as a continuous engineering discipline rather than an afterthought. The strategy combines programmatic checks, calibrated LLM-as-a-judge evaluators, and human review to detect subtle quality regressions. By inspecting intermediate agentic execution traces and real-world failure modes, teams ensure generative models remain reliable and aligned with product requirements. ### [Agent platform (Part 1): How we help Grab build and run AI agents at scale](https://yomu.fyi/post/agent-platform-part-1-how-we-help-grab-build-and-run-ai-agents-at-scal.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Kendrick Tan - Published: Jul 24, 2026 Grab scaled its internal AI tooling by transforming the infrastructure behind a single technical support bot into a standardized agent framework named LLM-Kit. The original Go-based bot revealed critical bottlenecks, including the lack of automated evaluation, painful provider migrations, fragmented observability, and excessive production scaffolding overhead. To address these frictions, engineers developed a FastAPI template pre-wired with LangGraph ReAct agents, remote Model Context Protocol servers, and built-in ROUGE, BLEU, and LLM-as-judge evaluation suites. The framework centralizes model access through an OpenAI-compatible GrabGPT Gateway and unifies observability across services, outbound calls, and agent steps using OpenTelemetry. This extracted scaffolding reduced day-one production setup time from two weeks to roughly an hour while supporting hundreds of production agents. ### [Building Service Topology at Scale: Architecture, Challenges, and Lessons Learned](https://yomu.fyi/post/building-service-topology-at-scale-architecture-challenges-and-lessons.md) - Company: [Netflix](https://yomu.fyi/company/netflix.md) - Author: Netflix Technology Blog - Published: Jul 13, 2026 Engineers at Netflix required a unified, real-time view of service dependencies to navigate distributed architecture and improve incident troubleshooting. Traditional batch systems introduced stale data, so the team created a streaming-first platform backed by reactive streams and backpressure handling to ingest flow records from multi-region Kafka streams and Server-Sent Events without data loss. The architecture partitions data into physically separate graph and columnar storage layers covering eBPF network flows, IPC metrics, and distributed traces. Network flow ingestion relies on a three-stage distributed aggregation pipeline using consistent hashing to resolve network intermediaries into logical application connections. The resulting production system serves time-travel and topology queries with sub-second latency while continuously updating dependency views. ### [Palana (Part 2): Architecting isolation, identity, and auditability for AI agents](https://yomu.fyi/post/palana-part-2-architecting-isolation-identity-and-auditability-for-ai.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Kevin Littlejohn - Published: Jun 21, 2026 Grab's Palana platform provisions isolated, Kubernetes-native runtime environments for autonomous AI agents using dedicated per-agent namespaces and role-based access controls. The architecture separates network enforcement across layers, applying Layer 3 and Layer 4 containment with Cilium and NetworkPolicy alongside Layer 7 application filtering evaluated by Open Policy Agent. Agent interactions with large language models route through a LiteLLM proxy wrapper that retrieves credentials from HashiCorp Vault based on Kubernetes pod context rather than client headers. Secrets management is divided between directly readable agent paths and proxy-only placeholder paths that prevent raw tokens from residing in runtime filesystems. An automated reaper monitors multi-source activity signals to shut down idle compute resources while preserving persistent storage and configuration state. ### [The Data Canary: How Netflix Validates Catalog Metadata](https://yomu.fyi/post/the-data-canary-how-netflix-validates-catalog-metadata.md) - Company: [Netflix](https://yomu.fyi/company/netflix.md) - Author: Netflix Technology Blog - Published: Jun 19, 2026 A manual mitigation action during an incident corrupted a data feed for a subset of titles, causing playback issues and catalog service failures that existing code canary systems failed to catch. To protect streaming reliability, Netflix built an automated data canary system that validates transformed catalog metadata prior to publication. The architecture utilizes a dedicated orchestrator alongside permanent baseline and canary service clusters to coordinate validation using real production traffic. By leveraging custom chaos experiment thresholds, sticky session affinity, and Starts Per Second playback metrics, the system detects regressions in under ten minutes and blocks publication automatically. Controlled failure injection experiments routing approximately 0.2% of global traffic confirmed that issues could be identified in 2.5 to 4 minutes. ### [Data Mesh at Grab (Part II): The foundational tools behind certification](https://yomu.fyi/post/data-mesh-at-grab-part-ii-the-foundational-tools-behind-certification.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Aezo Teo - Published: Apr 30, 2026 Grab operationalizes its Signals Marketplace data mesh through integrated platforms designed for continuous data certification and observability. The central metadata management platform, Hubble, extends open-source DataHub to model metadata as an event-driven graph and expose search, lineage, ownership, and data contracts. An automated certification engine built on the DataHub Actions framework continuously evaluates metadata changes, classifying assets into Uncertified, Certified, CertifiedPlus, or Revoked states. Genchi serves as the data quality observability layer, using Temporal and Kafka to run checks for freshness, volume completeness, schema stability, and semantic rules. To eliminate false-positive alerts caused by decoupled cron schedules, Genchi integrates with the Lighthouse monitoring service to trigger quality tests immediately upon pipeline completion. ### [How GitHub uses eBPF to improve deployment safety](https://yomu.fyi/post/how-github-uses-ebpf-to-improve-deployment-safety.md) - Company: [Github](https://yomu.fyi/company/github.md) - Author: Lawrence Gripper - Published: Apr 16, 2026 Deployment scripts can introduce dangerous circular dependencies when they rely on services or assets from platforms that are currently experiencing outages. Blocking network access at the host level is impractical because stateful nodes continue serving live traffic during rolling deployments. To solve this, GitHub isolates deploy scripts into dedicated Linux cGroups and attaches custom eBPF programs via the cilium/ebpf Go library. The system uses socket-address hooks to redirect DNS queries to a userspace proxy that checks a domain blocklist, while egress packet hooks map DNS transaction IDs to process IDs. This approach successfully prevents deploy-time circular dependencies, provides full command-line audit logs for blocked requests, and speeds up incident recovery. ### [From Custom to Open: Scalable Network Probing and HTTP/3 Readiness with Prometheus](https://yomu.fyi/post/from-custom-to-open-scalable-network-probing-and-http-3-readiness-with.md) - Company: [Slack](https://yomu.fyi/company/slack.md) - Author: Carlo Preciado - Published: Mar 31, 2026 Slack encountered a lack of client-side observability when introducing HTTP/3 at its edge. Because HTTP/3 runs over QUIC on UDP instead of TCP, neither existing commercial SaaS tools nor the internal Prometheus Blackbox Exporter could probe the new endpoints. To address this gap, an engineering intern integrated the open-source quic-go library into Prometheus Blackbox Exporter to build a native, configurable HTTP/3 probe. Slack deployed an internal implementation using the upstream functionality while waiting for pull request reviews, ultimately landing the contribution upstream. The resulting setup provides a unified view of HTTP/1.1, HTTP/2, and HTTP/3 metrics within Grafana, enabling reliable alerting and faster correlation with other telemetry. ### [Demystifying user journeys: Revolutionizing troubleshooting with auto tracking](https://yomu.fyi/post/demystifying-user-journeys-revolutionizing-troubleshooting-with-auto-t.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Alex Prokofiev - Published: Dec 23, 2025 Tracking user journeys across hundreds of mobile UI components via manual clickstream instrumentation frequently caused data gaps and prolonged issue resolution. Grab designed AutoTrack, an SDK that automatically records application lifecycle states, launch triggers, user interactions, and screen contexts. The system monitors transitions across foreground, background, and inactive states alongside entry points like deep links and push notifications. On the client side, AutoTrack captures complete screen hierarchy metadata from native Android activities, fragments, and iOS view controllers, redacting sensitive data to comply with privacy regulations. This automated trace enables engineering teams to reproduce elusive bugs, route incidents to domain owners, and generate automated UI test cases from real-world usage patterns. ### [Real-time data quality monitoring: Kafka stream contracts with syntactic and semantic test](https://yomu.fyi/post/real-time-data-quality-monitoring-kafka-stream-contracts-with-syntacti.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Yuanzhe Liu - Published: Nov 26, 2025 Kafka streams often suffer from syntactic and semantic data quality issues that propagate undetected to downstream consumers without real-time validation. Grab addressed this challenge by developing a standardized contract testing and observability framework within its Coban platform. Stakeholders define schema rules and field-level semantic validations, which can be recommended using large language models and anonymized sample data. A transformation engine converts these contracts into inverse SQL queries executed continuously by a FlinkSQL Test Runner on a dedicated consumer group. Problematic records are published to an alert topic, archived to AWS S3, and surfaced via Slack notifications and UI field-highlighting across more than 100 critical Kafka topics. ### [Improving Hugo stability and addressing oncall challenges through automation](https://yomu.fyi/post/improving-hugo-stability-and-addressing-oncall-challenges-through-auto.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Shuguang Xiang - Published: Mar 20, 2025 Grab's Hugo platform manages over 4,000 data ingestion pipelines for the company's data lake, but recurring failures caused significant on-call strain and data downtime due to complex triage processes. To resolve these operational challenges, an automated architecture was introduced featuring signal collection, automated diagnosis, root-cause tracking, auto-resolution, and a central dashboard. Rather than parsing extensive execution logs from Spark or Airflow, the diagnosis engine analyzes direct computation signals and temporal execution steps in parallel to pinpoint assignees and root causes. An asynchronous auto-resolution framework executes custom recovery handlers, such as automated backoff retries during database replica lags. This automated workflow reduced on-call workloads, improved dataset visibility, and shortened issue triage times across Grab's data ecosystem. [Older posts](https://yomu.fyi/topic/observability/page/2.md)