# Reliability
> 61 posts about Reliability, summarised, each linking to the original.

## Articles

### [Data Mesh at Grab (Part II): The foundational tools behind certification](https://yomu.fyi/post/data-mesh-at-grab-part-ii-the-foundational-tools-behind-certification.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Aezo Teo
- Published: Apr 30, 2026

Grab operationalizes its Signals Marketplace data mesh through integrated platforms designed for continuous data certification and observability. The central metadata management platform, Hubble, extends open-source DataHub to model metadata as an event-driven graph and expose search, lineage, ownership, and data contracts. An automated certification engine built on the DataHub Actions framework continuously evaluates metadata changes, classifying assets into Uncertified, Certified, CertifiedPlus, or Revoked states. Genchi serves as the data quality observability layer, using Temporal and Kafka to run checks for freshness, volume completeness, schema stability, and semantic rules. To eliminate false-positive alerts caused by decoupled cron schedules, Genchi integrates with the Lighthouse monitoring service to trigger quality tests immediately upon pipeline completion.


### [How GitHub uses eBPF to improve deployment safety](https://yomu.fyi/post/how-github-uses-ebpf-to-improve-deployment-safety.md)
- Company: [Github](https://yomu.fyi/company/github.md)
- Author: Lawrence Gripper
- Published: Apr 16, 2026

Deployment scripts can introduce dangerous circular dependencies when they rely on services or assets from platforms that are currently experiencing outages. Blocking network access at the host level is impractical because stateful nodes continue serving live traffic during rolling deployments. To solve this, GitHub isolates deploy scripts into dedicated Linux cGroups and attaches custom eBPF programs via the cilium/ebpf Go library. The system uses socket-address hooks to redirect DNS queries to a userspace proxy that checks a domain blocklist, while egress packet hooks map DNS transaction IDs to process IDs. This approach successfully prevents deploy-time circular dependencies, provides full command-line audit logs for blocked requests, and speeds up incident recovery.


### [Managing context in long-run agentic applications](https://yomu.fyi/post/managing-context-in-long-run-agentic-applications.md)
- Company: [Slack](https://yomu.fyi/company/slack.md)
- Author: Dominic Marks
- Published: Apr 13, 2026

Long-running multi-agent systems struggle with context management because accumulating raw message histories degrades inference quality, increases latency, and exceeds context window limits. In a collaborative security investigation platform, passing unrestricted history can also introduce confirmation bias across specialized agents. To maintain coherence across unbounded rounds of investigation, the system eliminates raw message history carryover between invocations. Instead, it coordinates agents through three structured context channels: a Director's Journal for orchestration memory, a Critic's Review that scores findings to filter hallucinations, and a Critic's Timeline of validated chronological events. This architecture provides agents with tailored context without overwhelming their inference capacity.


### [Improving storage efficiency in Magic Pocket, our immutable blob store](https://yomu.fyi/post/improving-storage-efficiency-in-magic-pocket-our-immutable-blob-store.md)
- Company: [Dropbox](https://yomu.fyi/company/dropbox.md)
- Author: Ilya Yakovlev,Andrew Cheung,Binoy Dash,Simran Jumani,Dmitriy Meyerzon,Mark Breitenbach,Ishan Mishra,Kazuaki Okumura,Mike White,Kevin Altschuler,Facundo Agriel
- Published: Apr 2, 2026

After introducing a direct erasure-coding service, Dropbox encountered severe storage fragmentation in its immutable blob store, Magic Pocket, due to a large number of under-filled volumes. Because their legacy compaction strategy was only designed to top off near-full volumes, the team created new multi-strategy compaction methods to aggressively consolidate sparse volumes. These approaches leverage dynamic programming and streaming pipelines to rapidly reclaim unused disk capacity at exabyte scale.


### [From Custom to Open: Scalable Network Probing and HTTP/3 Readiness with Prometheus](https://yomu.fyi/post/from-custom-to-open-scalable-network-probing-and-http-3-readiness-with.md)
- Company: [Slack](https://yomu.fyi/company/slack.md)
- Author: Carlo Preciado
- Published: Mar 31, 2026

Slack encountered a lack of client-side observability when introducing HTTP/3 at its edge. Because HTTP/3 runs over QUIC on UDP instead of TCP, neither existing commercial SaaS tools nor the internal Prometheus Blackbox Exporter could probe the new endpoints. To address this gap, an engineering intern integrated the open-source quic-go library into Prometheus Blackbox Exporter to build a native, configurable HTTP/3 probe. Slack deployed an internal implementation using the upstream functionality while waiting for pull request reviews, ultimately landing the contribution upstream. The resulting setup provides a unified view of HTTP/1.1, HTTP/2, and HTTP/3 metrics within Grafana, enabling reliable alerting and faster correlation with other telemetry.


### [Demystifying user journeys: Revolutionizing troubleshooting with auto tracking](https://yomu.fyi/post/demystifying-user-journeys-revolutionizing-troubleshooting-with-auto-t.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Alex Prokofiev
- Published: Dec 23, 2025

Tracking user journeys across hundreds of mobile UI components via manual clickstream instrumentation frequently caused data gaps and prolonged issue resolution. Grab designed AutoTrack, an SDK that automatically records application lifecycle states, launch triggers, user interactions, and screen contexts. The system monitors transitions across foreground, background, and inactive states alongside entry points like deep links and push notifications. On the client side, AutoTrack captures complete screen hierarchy metadata from native Android activities, fragments, and iOS view controllers, redacting sensitive data to comply with privacy regulations. This automated trace enables engineering teams to reproduce elusive bugs, route incidents to domain owners, and generate automated UI test cases from real-world usage patterns.


### [Data mesh at Grab part I: Building trust through certification](https://yomu.fyi/post/data-mesh-at-grab-part-i-building-trust-through-certification.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Chun Rong Phang
- Published: Aug 19, 2025

Rapid business growth across multiple verticals led Grab's centralized data engineering model to become an unscalable bottleneck, resulting in duplicate pipelines, ambiguous ownership, and broken downstream dependencies. To resolve these issues, the organization initiated a data mesh journey called Signals Marketplace that decentralizes data management and treats data as a product. A central data certification system establishes formal data contracts covering schemas, SLAs, freshness, and retention, while assigning clear Business Data Owners and Technical Data Owners. Breaches in contract guarantees automatically generate Data Production Incident tickets to enforce accountability and root-cause fixes. Consequently, 75% of internal queries now target certified assets, redundant tables saw a 400% year-over-year deprecation increase, and the total number of top-used datasets dropped by over 58%.


### [Effortless enterprise authentication at Grab: Dex in action](https://yomu.fyi/post/effortless-enterprise-authentication-at-grab-dex-in-action.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Kah Wei Lee
- Published: May 23, 2025

Grab needed a centralised system to simplify identity management, satisfy audit requirements, and standardise authentication across internal and external tools like Databricks and Datadog. The engineering team selected OpenID Connect as their standard protocol and adopted Dex, an open-source CNCF identity aggregator. Dex acts as an intermediary between applications and multiple identity providers to issue standardised OIDC tokens. To secure service-to-service communication, Grab implemented token exchange with trusted peer relationships rather than relying on privileged service accounts. Dex also provides a kill-switch mechanism that can route authentication traffic to an alternate provider during identity provider outages.


### [Introducing the SOP-driven LLM agent frameworks](https://yomu.fyi/post/introducing-the-sop-driven-llm-agent-frameworks.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Fujiao Liu
- Published: Apr 25, 2025

Standard operating procedure (SOP) driven Large Language Model agent frameworks address common generative AI challenges such as hallucinations, non-standard output formats, and branching navigation errors. Workflows are represented as hierarchical trees where nodes encapsulate actions or decision points that can be created using a visual editor and annotated with explicit external function calls. Execution relies on a tripartite architecture consisting of a Depth-First Search planner module with backtracking, an adaptive worker agent that limits API exposure and compresses context, and a multilingual user agent. Supporting tools include a Graph Retrieval-Augmented Generation pipeline, a plugin system integrating Python and SQL, and a state stack for pausing workflows during human intervention. In production deployments for fraud and account takeover investigations, the framework automated up to 87% of cases while cutting handling times substantially.


### [Improving Hugo stability and addressing oncall challenges through automation](https://yomu.fyi/post/improving-hugo-stability-and-addressing-oncall-challenges-through-auto.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Shuguang Xiang
- Published: Mar 20, 2025

Grab's Hugo platform manages over 4,000 data ingestion pipelines for the company's data lake, but recurring failures caused significant on-call strain and data downtime due to complex triage processes. To resolve these operational challenges, an automated architecture was introduced featuring signal collection, automated diagnosis, root-cause tracking, auto-resolution, and a central dashboard. Rather than parsing extensive execution logs from Spark or Airflow, the diagnosis engine analyzes direct computation signals and temporal execution steps in parallel to pinpoint assignees and root causes. An asynchronous auto-resolution framework executes custom recovery handlers, such as automated backoff retries during database replica lags. This automated workflow reduced on-call workloads, improved dataset visibility, and shortened issue triage times across Grab's data ecosystem.


### [Turbocharging GrabUnlimited with Temporal](https://yomu.fyi/post/turbocharging-grabunlimited-with-temporal.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Michel Parreno
- Published: Dec 12, 2024

GrabUnlimited experienced scaling bottlenecks, corrupted membership states, and elevated production incidents after its subscriber base grew by over 1000%. The original architecture relied on Amazon SQS state machines, 5-minute Redis locks, and daily batch cron jobs that overwhelmed the database and lacked granular idempotency during upstream retries. To eliminate these failure modes, the engineering team migrated the core membership lifecycle to Temporal's workflow orchestration engine. Replacing batch cron jobs with Temporal Timers distributed renewal operations throughout the day, while matching workflow IDs prevented race conditions between renewals and cancellations. This architectural transition resolved database bottlenecks and yielded an 80% reduction in open production incidents.


### [No version left behind: Our epic journey of GitLab upgrades](https://yomu.fyi/post/no-version-left-behind-our-epic-journey-of-gitlab-upgrades.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Saurabh Vajpayee
- Published: May 3, 2024

Grab's self-hosted GitLab instance fell approximately 14 months behind official releases, serving thousands of engineers and monorepos reaching up to 39TB in total footprint. To address security updates and system stability, the team established a structured upgrade routine using Terraform, Packer, and Ansible across GitLab's 5,000-user reference architecture. They implemented staged deployments, upgraded stateful Gitaly nodes via in-place rotation, and resolved primary node clustering bottlenecks under Praefect. By utilizing official GitLab Upgrade Paths and parallelizing component deployments, the team caught up on 24 months of releases in 11 months and reduced release lag from 396 days down to 35 days.


### [Ensuring data reliability and observability in risk systems](https://yomu.fyi/post/ensuring-data-reliability-and-observability-in-risk-systems.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Yi Ni Ong
- Published: Apr 23, 2024

Grab's in-house risk management platform, GrabDefence, relies on ingesting large volumes of upstream data to drive real-time heuristic rules and data science models. To prevent data discrepancies and missing information from degrading fraud detection, the team implemented a real-time data observability pipeline. The solution uses Apache Flink SQL alongside custom table functions like JSONEXPLOAD to flatten nested JSON structures and aggregate metrics over 5-minute tumbling windows. Processed metrics are sent to Datadog, where counters are grouped by source stream and evaluated using anomaly detection algorithms. Alerts are routed directly to Slack, reducing anomaly detection latency from days or weeks down to within the same day or hour.


### [Kafka on Kubernetes: Reloaded for fault tolerance](https://yomu.fyi/post/kafka-on-kubernetes-reloaded-for-fault-tolerance.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Fabrice Harbulot
- Published: Dec 26, 2023

Grab's real-time data streaming platform, Coban, operates Kafka on AWS Elastic Kubernetes Service using Strimzi, allocating an entire EC2 worker node with NVMe instance store volumes to each broker. An initial architecture suffered from client connection errors, broken Network Load Balancer target groups, and zombie Persistent Volume Claims when worker nodes terminated. To achieve automated fault tolerance, the team integrated the AWS Node Termination Handler in Queue Processor mode with Auto Scaling lifecycle hooks, ensuring Kafka receives a SIGTERM to migrate partition leadership gracefully before shutdown. They also introduced the Kubernetes Cluster Autoscaler to dynamically provision replacement nodes during maintenance events and used the AWS Load Balancer Controller with TargetGroupBinding custom resources to dynamically update load balancer targets using IP mode.


### [Championing CyberSecurity: Grab's bug bounty programme in 2023](https://yomu.fyi/post/championing-cybersecurity-grab-s-bug-bounty-programme-in-2023.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Nathaniel Callens
- Published: Dec 19, 2023

Grab reviewed the performance and milestones of its security bug bounty programme for 2023, marking eight years of running the initiative. Operating quarterly campaigns through HackerOne alongside dedicated internal teams, the company processed approximately 1,000 vulnerability submissions over the year, including 400 reports during Q1 alone. Cumulative payouts to external security researchers surpassed $700,000 since the programme's 2015 launch. To adapt to an expanding service portfolio, Grab broadened its scope to encompass joint ventures and acquisitions while introducing diversified financial and recognition-based rewards.


### [Evolution of quality at Grab](https://yomu.fyi/post/evolution-of-quality-at-grab.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Abby Alcantara
- Published: Mar 31, 2023

As Grab expanded its superapp, manual QA sign-offs led to prolonged testing cycles and a peak in major production defects around 2019. To counter defect leakage and improve delivery pace, the engineering organization transitioned to a shift-left testing strategy across its software development lifecycle. Quality engineers began writing Given/When/Then acceptance tests prior to coding so that software engineers could execute them during development, accompanied by Definition of Ready and Definition of Done standards. The team also structured a multi-layer test strategy spanning unit tests, pipeline-executed UI component tests with mocked APIs, backend integration tests, and a custom internal end-to-end framework. Between 2019 and 2022, these changes reduced major and critical production issues by 60% and development-phase critical bugs by 40%.


### [Debugging High Latency Due to Context Leaks](https://yomu.fyi/post/debugging-high-latency-due-to-context-leaks.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Sourabh Suman
- Published: Jun 30, 2021

Market-Store, Grab's feature store for real-time machine learning features, experienced latency spikes from under 200 milliseconds to 2 seconds as traffic grew. Metrics and logs showed no direct correlation to API issues, but heap profiling with PPROF revealed continuously increasing memory held by child contexts. Further analysis tracked the leak to an update in Grab's open-source Async Library, which switched background contexts to uncancelled task contexts for worker runners. Because parent contexts maintained references to these uncancelled child contexts, the garbage collector could not reclaim their memory. This progressive memory exhaustion directly degraded API latency.


### [Our Journey to Continuous Delivery at Grab (Part 2)](https://yomu.fyi/post/our-journey-to-continuous-delivery-at-grab-part-2.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Sylvain Bougerel
- Published: May 10, 2021

Conveyor, an in-house continuous delivery system at Grab, introduces hermetic deployments by tracking application code alongside static and dynamic configuration parameters. This hermeticity guarantees that production releases use combinations of versioned artifacts previously verified in staging, preventing rollback incompatibilities. Conveyor replaces single multi-environment pipelines with decoupled pipelines, while automating cluster locking, deployment slot scheduling, release note generation, and canary monitoring with automated rollbacks. These automations reduced production deployment failure rates from 1.5% to an average of 0.3% over a three-month period. Grab also doubled the volume of production changes between 2018 and 2020 while saving more than 5,000 man-days of engineering effort in 2020.


### [Customer Support Workforce Routing](https://yomu.fyi/post/customer-support-workforce-routing.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Suman Anand
- Published: Feb 5, 2021

Grab replaced its third-party customer support routing software with an in-house workforce routing system for Livechat to gain better priority controls, bespoke configurations, and deeper analytics. The platform separates requests into distinct priority and business queues, using parallel workers that spend varied time slices dequeuing higher-priority issues like safety concerns. To prevent request starvation, workers operate out of sync across queue priority levels while dynamic queue limits cap incoming volume based on agent availability and performance. The system routes requests through an intermediate Agent Group layer, calculating eligibility scores from proficiency and concurrency metrics while managing per-agent locks to prevent over-allocation.


### [Pharos - Searching Nearby Drivers on Road Network at Scale](https://yomu.fyi/post/pharos-searching-nearby-drivers-on-road-network-at-scale.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Hao Wu
- Published: Dec 22, 2020

Ride-hailing allocation requires identifying nearby drivers based on actual routing distance and estimated time of arrival rather than straight-line haversine distance. Grab developed Pharos, a distributed in-memory spatial microservice that evaluates road networks to match fast-moving drivers with pick-up requests. The system partitions OpenStreetMap road graphs by city and vehicle vertical while indexing driver positions and edge-based nodes using Adaptive Radix Trees. To locate candidates, Pharos projects driver coordinates onto road segments as phantom nodes and runs Incremental Network Expansion to compute isochrone reaches. Read-write concurrency is managed by switching tree root snapshots, allowing concurrent reads while updates produce new roots for subsequent lookups.


[Newer posts](https://yomu.fyi/topic/reliability.md) · [Older posts](https://yomu.fyi/topic/reliability/page/3.md)