---
title: "Observability"
description: "251 posts about Observability, summarised, each linking to the original."
---

# Observability
> 251 posts about Observability, summarised, each linking to the original.

## Articles

### [New in PostgreSQL 14: What every developer should know](https://yomu.fyi/post/new-in-postgresql-14-what-every-developer-should-know.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: Gurjeet Singh
- Published: Nov 28, 2021
- Type: Explainer

PostgreSQL 14 introduces several enhancements to improve application performance, data manipulation, query troubleshooting, and database security. Developers can traverse nested JSON data structures using subscripting and store noncontiguous values using multirange data types in single operations. Applications relying on libpq can leverage client-side query pipelining to send multiple queries across high-latency networks without waiting for individual responses, while parallel execution now supports foreign data wrappers, parallel sequential scans, and materialized view refreshes. For observability, query IDs are integrated into live execution tracking via pg\_stat\_activity, EXPLAIN VERBOSE outputs, and log files. Security upgrades include predefined read-only roles such as pg\_read\_all\_data alongside SCRAM-SHA-256 enabled as the default authentication mechanism.


### [Supabase Reports and Metrics](https://yomu.fyi/post/supabase-reports-and-metrics.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: Div Arora
- Published: Jul 29, 2021
- Type: Announcement

Supabase introduced a dedicated Reports section and updated project home pages to deliver observability across hosted Postgres projects and complementary services. Because developers can run unconstrained database queries and custom workloads, monitoring both project-level service usage and underlying infrastructure metrics is essential. The new dashboard displays one month of historical data, including CPU and memory usage alongside request volumes for Database, Auth, and Storage APIs. To collect metrics across hundreds of thousands of targets without high latency, Supabase implemented a federated monitoring architecture running regional Prometheus instances that scrape local exporters and aggregate subsets into a central VictoriaMetrics instance. All project databases now also launch with the pg\_stat\_statements extension enabled for future query statistics.


### [Debugging High Latency Due to Context Leaks](https://yomu.fyi/post/debugging-high-latency-due-to-context-leaks.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Sourabh Suman
- Published: Jun 30, 2021
- Type: Incident / postmortem

Market-Store, Grab's feature store for real-time machine learning features, experienced latency spikes from under 200 milliseconds to 2 seconds as traffic grew. Metrics and logs showed no direct correlation to API issues, but heap profiling with PPROF revealed continuously increasing memory held by child contexts. Further analysis tracked the leak to an update in Grab's open-source Async Library, which switched background contexts to uncancelled task contexts for worker runners. Because parent contexts maintained references to these uncancelled child contexts, the garbage collector could not reclaim their memory. This progressive memory exhaustion directly degraded API latency.


### [Building a Hyper Self-Service, Distributed Tracing and Feedback System for Rule & Machine Learning (ML) Predictions](https://yomu.fyi/post/building-a-hyper-self-service-distributed-tracing-and-feedback-system.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Warren Zhou
- Published: May 24, 2021
- Type: Problem & solution

Grab's Trust, Identity, Safety, and Security team processes billions of daily rule and machine learning decisions for fraud detection, safety, and identity checks. Earlier logging approaches using plain text Kibana logs and the ActionTrace library lacked structured formats, dynamic entity customization, and fine-grained access controls. To resolve these limitations, the team built Archivist, a centralized tracing, statistics, and feedback system. Archivist ingests events through an SDK into Kafka streams, buffers and routes data into Elasticsearch indices and Amazon S3, and provides a role-based user portal. The platform handles 80 million daily logs across roughly 50 business scenarios, reducing scenario onboarding times from days to minutes.


### [Hooking Up Fly Metrics](https://yomu.fyi/post/hooking-up-fly-metrics.md)
- Company: [Fly.io](https://yomu.fyi/company/fly-io.md)
- Published: May 20, 2021
- Type: Tutorial

Fly apps include built-in Prometheus instrumentation, exposing automatically collected metrics through a public Prometheus endpoint and allowing applications to export custom metrics. The walkthrough connects an organization’s Fly Prometheus URL to Grafana with a Fly token, then explains dashboard variables and PromQL queries using counters, gauges, histograms, labels, irate(), rate(), increase(), sum(), and regional grouping. Examples cover network traffic, TCP connections, HTTP response rates by region, and response-time heatmaps built from histogram buckets ranging from 5ms to 120s. Application instrumentation is shown for Go, Python Flask, and Node.js, with scraping configured in fly.toml through a metrics port and path. The post concludes that metrics are currently free, with some tier free forever, although a future pricing plan is expected for ambitious usage.


### [Fly's Prometheus Metrics](https://yomu.fyi/post/fly-s-prometheus-metrics.md)
- Company: [Fly.io](https://yomu.fyi/company/fly-io.md)
- Published: May 13, 2021
- Type: Explainer

Fly describes the Prometheus-style metrics infrastructure behind its globally distributed micro-VM platform and explains why it favors metrics over traditional monitoring checks. The system collects counters and other measurements through HTTP /metrics endpoints, stores them in a clustered VictoriaMetrics (“Vicky”) deployment, and uses Telegraf, Vector, exporters, and Consul to connect sources and manage discovery. Within Firecracker instances, custom init, the Nomad driver, and a vsock carry node-exporter-type JSON statistics from guests for collection, while applications can expose their own exporters. Deployments can declare an exporter port and /metrics path in fly.toml, after which Fly routes those application metrics into Vicky alongside platform data. The resulting Prometheus API is available for Grafana queries, dashboards, and alerts, with labels such as app, region, host, and instance; the post argues that the same historical data supports anomaly detection, trending, and planning.


### [Returning 575 Terabytes of Storage Space to Our Users](https://yomu.fyi/post/returning-575-terabytes-of-storage-space-to-our-users.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Lucas Nelaupe
- Published: Feb 25, 2020
- Type: Problem & solution

Android Vitals data revealed that 15.7% of Grab users had less than 1GB of free device storage and uninstalled the app at 1.2 times the normal rate. To understand on-device storage consumption, the team instrumented session launches using the Android StorageManager API to collect binary size, cache folder size, and total footprint metrics. Analysis showed unusually large cache sizes driven by orphaned cache folders from discontinued third-party libraries, including an image library replacement of Picasso by Glide. An automated cleanup routine deployed in app updates purged legacy cache directories upon launch. This mechanism reclaimed 575 terabytes of junk data across more than 13 million devices, averaging 40MB per user.


### [Structured Logging: The Best Friend You’ll Want When Things Go Wrong](https://yomu.fyi/post/structured-logging-the-best-friend-you-ll-want-when-things-go-wrong.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Aditya Praharaj
- Published: Mar 5, 2019
- Type: Problem & solution

Grab redesigned its backend logging approach to address mounting vendor costs, query language limitations, and debugging difficulties across a growing microservices ecosystem. Most services previously emitted syslog-style key-value logs almost entirely at the INFO level, which made volume reduction difficult and lacked causal ordering and automated correlation. The engineering team migrated to a self-managed Elastic stack backend and built a structured logging library in Go from the ground up. This framework introduces dynamic log-level adjustment at runtime, automatic trace-based log correlation via Grab-Kit, and the Common Grab Log Schema to enforce consistent JSON formatting without Elasticsearch indexing conflicts.


### [Troubleshooting Unusual AWS ELB 5XX Error](https://yomu.fyi/post/troubleshooting-unusual-aws-elb-5xx-error.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Dharmarth Shah
- Published: May 10, 2017
- Type: Problem & solution

Grab experienced intermittent HTTP 5XX alerts when its Gothena service sent driver location updates to the Astrolabe service through an AWS Elastic Load Balancer (ELB). CloudWatch metrics revealed that requests were failing to reach healthy backend instances because of an uneven load distribution favoring a single ELB node in one Availability Zone. The team verified that Route 53 was properly using Alias records and ruled out OS-level DNS caching since Linux does not cache DNS queries by default. Connection inspection with netstat across multiple Go services confirmed a heavily skewed distribution of connections toward specific ELB IP addresses. Comparative tests with cURL, tcpdump, Go, Python, and Ruby in an isolated environment demonstrated that Go reused connections across requests while other runtimes opened new connections per request.


### [The Curious Case of the Phantom Instance](https://yomu.fyi/post/the-curious-case-of-the-phantom-instance.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Lian Yuanlin
- Published: Dec 28, 2015
- Type: Problem & solution

Datadog dashboards for the grab\_attention cluster displayed periodic 1.5X step increases in Elastic Load Balancer (ELB) health check requests and ElastiCache Redis connections, creating the illusion of an untracked instance running outside Auto Scaling Group records. Inspecting instance hostname tags revealed that the existing two instances were simply receiving elevated ping counts from the load balancers. AWS Support clarified that ELB scaling events provision new nodes while keeping old nodes running for roughly 90 minutes to handle cached DNS clients. Investigation also revealed that two separate ELBs were attached to the cluster, altering expected request baselines. Furthermore, the application's health check endpoint initiated a non-pooled Redis connection on every request, directly translating load balancer pings into database connection spikes.


### [Hutils – Explore your structured log data](https://yomu.fyi/post/hutils-explore-your-structured-log-data.md)
- Company: [Heroku](https://yomu.fyi/company/heroku.md)
- Author: Brandur Leach
- Published: Sep 5, 2014
- Type: Announcement

Heroku’s internal components use logfmt to capture production activity, enabling retrospective request analysis, flexible trace queries, historical metrics, and incident investigation. The format can also become an unreadable 51-page wall of text, motivating hutils, a small suite of tools installable through Rubygems for exploring structured log data. Its ltap command extracts message streams from Splunk or Papertrail using a search query, while lviz parses messages into a tree that groups common key/value nodes and reduces duplicate data; the tools are designed to chain together. Additional commands include lcut for selecting fields and lfmt for colorized, whitespace-enhanced formatting, with both also supporting heroku logs --tail.


[Newer posts](https://yomu.fyi/topic/observability/page/12.md)