# Monitoring > 18 posts about Monitoring, summarised, each linking to the original. ## Articles ### [OpenTelemetry Comes to IntelliJ IDEA, GoLand, PyCharm, and WebStorm](https://yomu.fyi/post/opentelemetry-comes-to-intellij-idea-goland-pycharm-and-webstorm.md) - Company: [Jetbrains](https://yomu.fyi/company/jetbrains.md) - Author: Egor Klimov - Published: Aug 26, 2026 With the 2026.2 release, JetBrains expanded its OpenTelemetry plugin from Rider to IntelliJ IDEA, GoLand, PyCharm, and WebStorm. The tool captures logs, metrics, traces, and service maps locally without requiring an external observability backend. Developers can search structured log records, plot metric values over time, inspect distributed spans, and verify communication paths across databases and message queues during local runs. To ingest telemetry, the plugin automatically configures OpenTelemetry Protocol environment variables for supported run configurations and terminal sessions, or accepts forwarded data from existing collectors. Additionally, experimental Model Context Protocol support allows AI coding agents to query gathered logs, spans, and service topology. ### [How we knew COVID was over (and what our models had to unlearn)](https://yomu.fyi/post/how-we-knew-covid-was-over-and-what-our-models-had-to-unlearn.md) - Company: [Airbnb](https://yomu.fyi/company/airbnb.md) - Author: Harrison Katz - Published: Aug 19, 2026 Airbnb's forecasting team outlines their framework for managing production models that experience drift or external shocks. Rather than treating model updates as a generic retraining task, they distinguish between refitting with new data, respecifying the underlying model structure, and intentionally holding without changes. This discipline prevents teams from chasing temporary noise, carrying obsolete crisis assumptions, or over-engineering solutions during disruptions. ### [BGP Role model: tracking the adoption of RFC 9234](https://yomu.fyi/post/bgp-role-model-tracking-the-adoption-of-rfc-9234.md) - Company: [Cloudflare](https://yomu.fyi/company/cloudflare.md) - Author: Bryton Herdes - Published: Aug 18, 2026 Border Gateway Protocol route leaks occur when routing announcements propagate beyond their intended customer-provider or peer relationships, causing traffic misdirection and performance degradation. RFC 9234 embeds relationship awareness directly into BGP by establishing negotiated BGP Roles and attaching an Only to Customer (OTC) transitive path attribute to prevent unauthorized route propagation. To assess real-world adoption, Cloudflare monitored OTC attribute transmission across peer networks and discovered that 33.1 percent of IPv4 and 17 percent of IPv6 paths were missing the OTC attribute. Investigation revealed that Tier-1 transit providers AS1299 and AS3257 were stripping the attribute due to legacy defensive BGP error-handling configurations. Following direct engagement, AS1299 updated its configurations to preserve OTC propagation, whereas AS3257 continued stripping it, highlighting the necessity of broader vendor and carrier support across the Internet. ### [Total eclipse of the Internet: traffic impacts in Iceland, Spain, and Portugal](https://yomu.fyi/post/total-eclipse-of-the-internet-traffic-impacts-in-iceland-spain-and-por.md) - Company: [Cloudflare](https://yomu.fyi/company/cloudflare.md) - Author: Sabina Zejnilovic - Published: Aug 13, 2026 On August 12, a total solar eclipse crossed the North Atlantic and Europe, passing directly over Iceland, northern Spain, and Portugal. Cloudflare Radar analyzed HTTP request volumes in five-minute intervals across affected countries, comparing the data against a baseline derived from the previous three Wednesdays. Traffic reductions aligned closely with peak solar obscuration, with totality regions recording drops between 15% and 30% and specific areas shifting by up to -46.7%. Once the sun reappeared, request volumes rebounded to normal levels within minutes as users resumed device activity. The observed traffic declines were driven by aggregate human behavior rather than network failures. ### [Certificate Transparency Monitoring is now generally available](https://yomu.fyi/post/certificate-transparency-monitoring-is-now-generally-available.md) - Company: [Cloudflare](https://yomu.fyi/company/cloudflare.md) - Author: Jenny Yang - Published: Aug 13, 2026 Cloudflare announced the general availability of Certificate Transparency Monitoring alongside a filtering mechanism to eliminate noisy alert emails. Previously, the monitoring system alerted domain owners for all Certificate Transparency log entries, including routine, automated renewals for Universal SSL, Advanced Certificate Manager, and backup certificates. Because the alerting flow and certificate ordering service operated independently, existing identifiers like TBSCertificate hashes arrived too late to verify pre-certificates. To resolve this race condition, the ordering service now computes an SHA-256 hash of the DER-encoded SubjectPublicKeyInfo structure at key generation. When the alerting service discovers a log entry, it recomputes this hash to verify ownership against the ordering database, suppressing alerts for Cloudflare-managed certificates while preserving alerts for external or unexpected issuances. ### [Cloudflare DDoS Threat Report H1 2026: 1 Tbps attacks soar as DNS floods and geopolitical tensions drive a new wave](https://yomu.fyi/post/cloudflare-ddos-threat-report-h1-2026-1-tbps-attacks-soar-as-dns-flood.md) - Company: [Cloudflare](https://yomu.fyi/company/cloudflare.md) - Author: Cloudforce One - Published: Aug 11, 2026 In the first half of 2026, network-layer DDoS attacks exceeding 1 Tbps surged by 519% quarter-over-quarter, with 935 mitigated attacks recorded across the period. Threat actors shifted their focus toward reflection and amplification methods, making DNS-based floods and CLDAP floods the dominant vectors. Despite the rise in hyper-volumetric traffic, 96.62% of network-layer attacks remained under 500 Mbps and 90.60% ended in under 10 minutes. Geopolitical conflicts and international events heavily influenced target selection, pushing the Media, Production & Publishing industry to the top position. Because brief multi-gigabit bursts trigger downstream routing instability and application timeouts before analysts can respond, automated, always-on edge mitigation has become critical. ### [From ranking to recommended: get your site ready to thrive in the age of AI agents](https://yomu.fyi/post/from-ranking-to-recommended-get-your-site-ready-to-thrive-in-the-age-o.md) - Company: [Cloudflare](https://yomu.fyi/company/cloudflare.md) - Author: Matthew Conroy - Published: Aug 6, 2026 Discoverability is shifting from traditional search engine rankings to recommendations generated by AI agents acting on behalf of users. To address this shift, Cloudflare integrated Agent Readiness diagnostics and an Answer Engine Optimization tool into its dashboard. The Diagnostics feature evaluates hostnames against machine-readable standards across tiers ranging from robots.txt and Markdown rendering to OAuth discovery and agent payment protocols. Simultaneously, the Answer Engine Optimization tool measures how frequently leading assistants like Claude and GPT cite, mention, and prominently position brands across industry categories. By evaluating prompt responses using Cloudflare AI Gateway and Workers AI alongside access logs, site owners can systematically track agent traffic, benchmark against competitors, and remediate crawl blockers. ### [From Custom to Open: Scalable Network Probing and HTTP/3 Readiness with Prometheus](https://yomu.fyi/post/from-custom-to-open-scalable-network-probing-and-http-3-readiness-with.md) - Company: [Slack](https://yomu.fyi/company/slack.md) - Author: Carlo Preciado - Published: Mar 31, 2026 Slack encountered a lack of client-side observability when introducing HTTP/3 at its edge. Because HTTP/3 runs over QUIC on UDP instead of TCP, neither existing commercial SaaS tools nor the internal Prometheus Blackbox Exporter could probe the new endpoints. To address this gap, an engineering intern integrated the open-source quic-go library into Prometheus Blackbox Exporter to build a native, configurable HTTP/3 probe. Slack deployed an internal implementation using the upstream functionality while waiting for pull request reviews, ultimately landing the contribution upstream. The resulting setup provides a unified view of HTTP/1.1, HTTP/2, and HTTP/3 metrics within Grafana, enabling reliable alerting and faster correlation with other telemetry. ### [Real-time data quality monitoring: Kafka stream contracts with syntactic and semantic test](https://yomu.fyi/post/real-time-data-quality-monitoring-kafka-stream-contracts-with-syntacti.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Yuanzhe Liu - Published: Nov 26, 2025 Kafka streams often suffer from syntactic and semantic data quality issues that propagate undetected to downstream consumers without real-time validation. Grab addressed this challenge by developing a standardized contract testing and observability framework within its Coban platform. Stakeholders define schema rules and field-level semantic validations, which can be recommended using large language models and anonymized sample data. A transformation engine converts these contracts into inverse SQL queries executed continuously by a FlinkSQL Test Runner on a dedicated consumer group. Problematic records are published to an alert topic, archived to AWS S3, and surfaced via Slack notifications and UI field-highlighting across more than 100 critical Kafka topics. ### [Building a Spark observability product with StarRocks: Real-time and historical performance analysis](https://yomu.fyi/post/building-a-spark-observability-product-with-starrocks-real-time-and-hi.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Huong Vuong - Published: Mar 6, 2025 Grab redesigned its Spark observability platform, Iris, to overcome limitations associated with its previous Telegraf, InfluxDB, and Grafana stack. InfluxDB presented operational challenges due to limited SQL compatibility, poor handling of string metadata, and query degradation on high-cardinality identifiers. The team replaced InfluxDB with StarRocks to serve as a unified analytical engine for both real-time cluster metrics and historical analysis. StarRocks ingests metrics directly from Kafka via routine load tasks, storing worker and Spark event data in partitioned duplicate-key OLAP tables linked by worker and application identifiers. This architecture eliminated intermediate ingestion agents, simplified S3 data lake backups, and enabled a custom web application alongside Superset for consistent querying. ### [Ensuring data reliability and observability in risk systems](https://yomu.fyi/post/ensuring-data-reliability-and-observability-in-risk-systems.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Yi Ni Ong - Published: Apr 23, 2024 Grab's in-house risk management platform, GrabDefence, relies on ingesting large volumes of upstream data to drive real-time heuristic rules and data science models. To prevent data discrepancies and missing information from degrading fraud detection, the team implemented a real-time data observability pipeline. The solution uses Apache Flink SQL alongside custom table functions like JSONEXPLOAD to flatten nested JSON structures and aggregate metrics over 5-minute tumbling windows. Processed metrics are sent to Datadog, where counters are grouped by source stream and evaluated using anomaly detection algorithms. Alerts are routed directly to Slack, reducing anomaly detection latency from days or weeks down to within the same day or hour. ### [Iris - Turning observations into actionable insights for enhanced decision making](https://yomu.fyi/post/iris-turning-observations-into-actionable-insights-for-enhanced-decisi.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Huong Vuong - Published: Apr 3, 2024 Standard infrastructure monitoring tools track CPU and memory usage at the host or container level, obscuring the precise resource consumption of individual Spark jobs on shared instances. To achieve granular observability, the Iris platform gathers job metrics directly at the Java Virtual Machine level without requiring changes to user application code. Data collection relies on Uber's JVM Profiler for deep JVM, thread, and memory pool diagnostics alongside sparkMeasure for stage and task execution statistics. Collected metrics route asynchronously through an Apache Kafka queue to avoid execution bottlenecks, feeding a Telegraf, InfluxDB, and Grafana stack for real-time monitoring while archiving to a data lake for offline analytics. This multi-tiered telemetry enables engineering teams to identify over- or under-provisioned jobs and optimize ETL resource allocations. ### [Android App Size at Scale with Project Bonsai](https://yomu.fyi/post/android-app-size-at-scale-with-project-bonsai.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Nguyen Van Minh - Published: Mar 1, 2024 Grab launched Project Bonsai to optimize the size of its Android superapp, which encompassed over four million lines of code across 1,500 modules. Because large app sizes negatively impact user acquisition on low-end devices with constrained bandwidth, the engineering team structured its optimization strategy around measurement, reduction, and containment. The team developed App Sizer, a custom CI-integrated tool that tracks binary compositions and module size contributions in Grafana. Initial reductions targeted Java and Kotlin dex files, resolving bloated transitive R classes by upgrading the Android Gradle Plugin to eliminate overly broad R8 retention rules. These initiatives achieved a 26% reduction in app download size while simultaneously decreasing overall disk footprint. ### [Using mobile sensor data to encourage safer driving](https://yomu.fyi/post/using-mobile-sensor-data-to-encourage-safer-driving.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Laiyi Lin - Published: Oct 25, 2022 Dedicated vehicular inertial measurement units are prohibitively expensive to install across large fleets, while customer feedback lacks the coverage required to assess driving safety comprehensively. To monitor driver behavior at scale, Grab processes smartphone sensor data collected via a mobile telematics SDK at high sampling frequencies. A custom orientation algorithm calculates Euler roll and pitch angles from gravity and resolves yaw through principal component analysis to align phone measurements with vehicle axes. The pipeline applies low-pass or Kalman filtering to handle varying noise levels, followed by threshold peak-finding algorithms to detect harsh acceleration, braking, and cornering. Speeding events are identified by comparing GPS speeds against known map limits or aggregated speed profiles derived from free-flowing traffic. ### [How We Built a Logging Stack at Grab](https://yomu.fyi/post/how-we-built-a-logging-stack-at-grab.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Daniel Kasen - Published: Jul 31, 2019 Grab needed a scalable logging platform to replace slow, fragmented systems that hindered debugging across their growing service fleet. Generating 25TB of daily logs, the team built a horizontally scalable Elasticsearch cluster configured via Ansible and monitored with Datadog. Although the initial proof of concept assigned all node roles (ingest, coordinator, master, and data) to every machine, operating at scale introduced major challenges with JVM heap exhaustion and cluster stability. The team resolved memory pressure and performance bottlenecks by tuning circuit breakers, lowering field data cache limits, adjusting shard allocations based on segment memory, and disabling translog compression during shard transfers. ### [Introducing Grab-Kit: Distributed Service Design at Grab](https://yomu.fyi/post/introducing-grab-kit-distributed-service-design-at-grab.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Karen Kue - Published: Jun 8, 2018 As Grab migrated from a monolith to microservices, maintaining consistency, coordination, and code quality across rapidly expanding teams became a major engineering challenge. To address this, the Developer Experience team built Grab-Kit, a Go framework that automates service scaffolding, code generation, and distributed system design patterns. The framework uses Protocol Buffer definition files as a single source of truth to generate data transfer objects, communication bindings, and standardized middleware for logging and profiling. Grab-Kit also features declarative metrics definitions that synchronize with the DataDog API to build and update service dashboards automatically. Adopting the framework reduced development time for creating new services by up to 70% in teams such as GrabFood while improving overall system stability. ### [Troubleshooting Unusual AWS ELB 5XX Error](https://yomu.fyi/post/troubleshooting-unusual-aws-elb-5xx-error.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Dharmarth Shah - Published: May 10, 2017 Grab experienced intermittent HTTP 5XX alerts when its Gothena service sent driver location updates to the Astrolabe service through an AWS Elastic Load Balancer (ELB). CloudWatch metrics revealed that requests were failing to reach healthy backend instances because of an uneven load distribution favoring a single ELB node in one Availability Zone. The team verified that Route 53 was properly using Alias records and ruled out OS-level DNS caching since Linux does not cache DNS queries by default. Connection inspection with netstat across multiple Go services confirmed a heavily skewed distribution of connections toward specific ELB IP addresses. Comparative tests with cURL, tcpdump, Go, Python, and Ruby in an isolated environment demonstrated that Go reused connections across requests while other runtimes opened new connections per request. ### [The Curious Case of the Phantom Instance](https://yomu.fyi/post/the-curious-case-of-the-phantom-instance.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Lian Yuanlin - Published: Dec 28, 2015 Datadog dashboards for the grab\_attention cluster displayed periodic 1.5X step increases in Elastic Load Balancer (ELB) health check requests and ElastiCache Redis connections, creating the illusion of an untracked instance running outside Auto Scaling Group records. Inspecting instance hostname tags revealed that the existing two instances were simply receiving elevated ping counts from the load balancers. AWS Support clarified that ELB scaling events provision new nodes while keeping old nodes running for roughly 90 minutes to handle cached DNS clients. Investigation also revealed that two separate ELBs were attached to the cluster, altering expected request baselines. Furthermore, the application's health check endpoint initiated a non-pooled Redis connection on every request, directly translating load balancer pings into database connection spikes.