---
title: "Reliability"
description: "289 posts about Reliability, summarised, each linking to the original."
---

# Reliability
> 289 posts about Reliability, summarised, each linking to the original.

## Articles

### [Ensuring data reliability and observability in risk systems](https://yomu.fyi/post/ensuring-data-reliability-and-observability-in-risk-systems.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Yi Ni Ong
- Published: Apr 23, 2024
- Type: Problem & solution

Grab's in-house risk management platform, GrabDefence, relies on ingesting large volumes of upstream data to drive real-time heuristic rules and data science models. To prevent data discrepancies and missing information from degrading fraud detection, the team implemented a real-time data observability pipeline. The solution uses Apache Flink SQL alongside custom table functions like JSONEXPLOAD to flatten nested JSON structures and aggregate metrics over 5-minute tumbling windows. Processed metrics are sent to Datadog, where counters are grouped by source stream and evaluated using anomaly detection algorithms. Alerts are routed directly to Slack, reducing anomaly detection latency from days or weeks down to within the same day or hour.


### [Kafka on Kubernetes: Reloaded for fault tolerance](https://yomu.fyi/post/kafka-on-kubernetes-reloaded-for-fault-tolerance.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Fabrice Harbulot
- Published: Dec 26, 2023
- Type: Problem & solution

Grab's real-time data streaming platform, Coban, operates Kafka on AWS Elastic Kubernetes Service using Strimzi, allocating an entire EC2 worker node with NVMe instance store volumes to each broker. An initial architecture suffered from client connection errors, broken Network Load Balancer target groups, and zombie Persistent Volume Claims when worker nodes terminated. To achieve automated fault tolerance, the team integrated the AWS Node Termination Handler in Queue Processor mode with Auto Scaling lifecycle hooks, ensuring Kafka receives a SIGTERM to migrate partition leadership gracefully before shutdown. They also introduced the Kubernetes Cluster Autoscaler to dynamically provision replacement nodes during maintenance events and used the AWS Load Balancer Controller with TargetGroupBinding custom resources to dynamically update load balancer targets using IP mode.


### [Championing CyberSecurity: Grab's bug bounty programme in 2023](https://yomu.fyi/post/championing-cybersecurity-grab-s-bug-bounty-programme-in-2023.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Nathaniel Callens
- Published: Dec 19, 2023
- Type: Explainer

Grab reviewed the performance and milestones of its security bug bounty programme for 2023, marking eight years of running the initiative. Operating quarterly campaigns through HackerOne alongside dedicated internal teams, the company processed approximately 1,000 vulnerability submissions over the year, including 400 reports during Q1 alone. Cumulative payouts to external security researchers surpassed $700,000 since the programme's 2015 launch. To adapt to an expanding service portfolio, Grab broadened its scope to encompass joint ventures and acquisitions while introducing diversified financial and recognition-based rewards.


### [Abstraction Engineering](https://yomu.fyi/post/abstraction-engineering.md)
- Company: [Ramp](https://yomu.fyi/company/ramp.md)
- Author: Rodda John, Jared Wasserman
- Published: Oct 31, 2023
- Type: Explainer

Ramp’s early product encoded opinionated limits, but by fall 2022 customers needed configuration based on departments, accounting fields, subsidiaries, budgets, and HRIS fields, making incremental feature work increasingly complex. The team reframed these requests as a generic workflows platform in which Python functions served as actions, boolean logic as conditions, and workflows as graphs persisted in Postgres. After rejecting a dependency-based design, they built a graph-traversal engine, a Postgres-backed queue that retries asynchronous tasks, and an SDK for expressing constructs such as elif and else. The infrastructure now supports customer-facing policies and internal tasks; more than 45 million workflows have run, at over a million per day, while the core engine has not changed since launch. The project took weeks upfront but saved the engineering organization months by turning complex features into applications of broader abstractions.


### [Multiple Logs for Resiliency](https://yomu.fyi/post/multiple-logs-for-resiliency.md)
- Company: [Fly.io](https://yomu.fyi/company/fly-io.md)
- Published: Jul 21, 2023
- Type: Problem & solution

An unexpected network outage can make an application misbehave while also removing crucial log data from the normal Fly.io pipeline. The post traces that pipeline from application STDOUT through Fly.io, NATS, the Log Shipper, Vector, and a third-party provider, then presents two ways to add redundancy. One strategy broadcasts framework logs to a separate file on each machine, preferably on a persistent volume with rotation; a Rails example uses ActiveSupport::Logger.broadcast. The other runs a separate app that subscribes directly to NATS, parses messages, and writes selected fields to disk, with a Fly authentication token and provisions for reconnection and error handling. File copies are more failsafe but raw and harder to fetch, while a custom shipper can produce more usable data but is less bulletproof.


### [More Predictable Shared Dyno Performance](https://yomu.fyi/post/more-predictable-shared-dyno-performance.md)
- Company: [Heroku](https://yomu.fyi/company/heroku.md)
- Author: Michael Friis
- Published: Apr 11, 2023
- Type: Announcement

Heroku describes a year-long effort to make CPU access more predictable for shared dynos in the Common Runtime. Previously, individual dynos could burst CPU use relatively freely when capacity was available, improving utilization but sometimes allowing misconfigured or malicious “noisy neighbors” to cause response-time spikes or H12 timeouts. The platform introduced improved resource isolation for eco, basic, standard-1X, and standard-2X Dynos, limiting bursts while preserving the ability to burst and aiming for fairer access to shared CPU resources. The change does not alter the number of dynos per instance, and Performance, Private, and Shield type Dynos are excluded because they run on dedicated instances.


### [Adventures in Garbage Collection: Improving GC Performance in our Massive Monolith](https://yomu.fyi/post/adventures-in-garbage-collection-improving-gc-performance-in-our-massi.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Apr 6, 2023
- Type: Problem & solution

Shopify investigated high Ruby garbage-collection latency in its monolith, where minor marks took about 3 ms but major marks could exceed four seconds and some requests spent multiple seconds in GC. Logging showed that roughly 75% of major marks were triggered by oldmalloc and nearly 25% by shady, while nofree became dominant after oldmalloc was addressed. The team used production experiments, improved metrics, changed GC environment settings, increased initial heap slots, and added out-of-band collection during request processing. A 128 GB oldmalloc threshold reduced major marking by about 20%; later out-of-band collection cut the P99.99 and P99.9 GC-time tail by almost tenfold and largely eliminated median-request pauses. The resulting configuration, with collection frequency adapting from every 128 to every 512 requests, was deployed across the fleet after about three weeks.


### [Evolution of quality at Grab](https://yomu.fyi/post/evolution-of-quality-at-grab.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Abby Alcantara
- Published: Mar 31, 2023
- Type: Problem & solution

As Grab expanded its superapp, manual QA sign-offs led to prolonged testing cycles and a peak in major production defects around 2019. To counter defect leakage and improve delivery pace, the engineering organization transitioned to a shift-left testing strategy across its software development lifecycle. Quality engineers began writing Given/When/Then acceptance tests prior to coding so that software engineers could execute them during development, accompanied by Definition of Ready and Definition of Done standards. The team also structured a multi-layer test strategy spanning unit tests, pipeline-executed UI component tests with mocked APIs, backend integration tests, and a custom internal end-to-end framework. Between 2019 and 2022, these changes reduced major and critical production issues by 60% and development-phase critical bugs by 40%.


### [The Case Against Monkey Patching, From a Rails Core Team Member](https://yomu.fyi/post/the-case-against-monkey-patching-from-a-rails-core-team-member.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Feb 21, 2023
- Type: Explainer

The post argues that Ruby monkey patches—global changes to existing objects or libraries—should be used sparingly because they are brittle, dangerous, and often unnecessary. They can make Rails and Ruby upgrades harder by depending on private APIs, leave applications exposed to vulnerabilities if security fixes are not mirrored, and accumulate poorly documented technical debt. One Shopify example, activerecord-pedant-adapter, changes Active Record’s MySQL2Adapter so database connections report MySQL query warnings; upstreaming that behavior into Rails allowed the gem to be archived and the patch removed. The recommended alternative is to investigate whether an upgrade or documentation resolves the issue, fix genuine defects upstream, and, when patching is unavoidable, document and test it while maintaining a removal plan.


### [Type Constraints in 65 lines of SQL](https://yomu.fyi/post/type-constraints-in-65-lines-of-sql.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: oli\_rice
- Published: Feb 17, 2023
- Type: Tutorial

PostgreSQL’s extensible type system can encode validation and presentation behavior for a custom Semantic Versioning (SemVer) type. The article stores major, minor, patch, pre-release, and build-metadata components in a composite type, then wraps that type in a domain whose check constraint rejects null or negative numeric components and invalid array elements. A SQL helper validates nullable arrays with regular expressions, and the domain automatically enforces these rules when values are inserted into a table. For ergonomics, a function formats the composite value as a SemVer string, while a stored generated column persists that representation; the article also notes possible parsers, operators, and aggregation functions. It concludes that the ceremony is worthwhile when data integrity is critical and the specification is stable or changes infrequently.


### [The Hardest Part of Writing Tests is Getting Started](https://yomu.fyi/post/the-hardest-part-of-writing-tests-is-getting-started.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Feb 2, 2023
- Type: Tutorial

The article presents getting started with automated testing as the main obstacle, especially when a full suite requires helper tools, stubs, and fixtures that can overwhelm beginners. It argues that tests serve as executable documentation, preserve intended behavior and edge cases, and expose regressions before changes reach production. The recommended progression is to install a framework and verify both passing and failing tests, cover isolated functions, add stubs with sanitized API-response fixtures, and then build end-to-end tests while monitoring coverage. In an ETL example, API stubbing avoided expensive, slow calls, while database checks validated loaded and transformed data; the resulting tests covered more than 80 percent of the codebase and saved time for the author and team. The article also proposes E2E tests as a way to safely approach legacy code before refactoring it.


### [Performance Testing At Scale—for BFCM and Beyond](https://yomu.fyi/post/performance-testing-at-scale-for-bfcm-and-beyond.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Jan 27, 2023
- Type: Benchmark

Shopify treats Black Friday Cyber Monday (BFCM) as its most critical period, and uses full-scale BFCM Scale Testing to assess whether the platform can handle peak traffic. Preparation combines capacity planning, resiliency testing, single-application performance testing, and repeated exercises that scale core services to their BFCM profiles before generating load. The hybrid architecture centers on Shopify Core, a monolithic Ruby application supported by services for storefront rendering, analytics, fraud analysis, payments, webhooks, and more. Five load flows cover browsing and buying, admin activity, flash sales, Storefront APIs, and Hydrogen and Oxygen, with Cronograma layering requests gradually and providing an abort switch. The exercises surfaced connection-count pressures, a Kubernetes ingress-nginx endpoint limit, and uneven load-balancer distribution; mitigations included refactoring, scaling dependent services, increasing pod capacity, and dedicated nginx node pools.


### [Catching Anomalies Early in Mobile App Releases](https://yomu.fyi/post/catching-anomalies-early-in-mobile-app-releases.md)
- Company: [Nextdoor](https://yomu.fyi/company/nextdoor.md)
- Author: Walt Leung
- Published: Jan 11, 2023
- Type: Problem & solution

Nextdoor deploys weekly mobile updates across iOS and Android using phased rollouts starting at 1% adoption to minimize blast radius. Standard aggregate observability cannot reliably detect early regressions because early adopters skew significantly more active than average users, obscuring silent drops in overall metric noise. To overcome this selection bias, Nextdoor built App Release Anomaly Detection using difference-in-differences causal inference. The methodology verifies pre-adoption trends with standard deviation bounds and fits a linear regression model to estimate the effect against an unobserved counterfactual. During a rollout of iOS version v1.234.5, this model identified a statistically significant app session decline at 1% adoption, isolating the regression 10 days earlier than week-over-week metrics.


### [Automatically Rotating GitHub Tokens (So You Don’t Have To)](https://yomu.fyi/post/automatically-rotating-github-tokens-so-you-don-t-have-to.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Dec 20, 2022
- Type: Problem & solution

Shopify needed automated GitHub workflows that could clone from private repositories, update organization-level projects, and trigger downstream workflows but could not use GITHUB\_TOKEN. Personal access tokens supplied those capabilities, yet introduced developer churn, user-attributed actions, and no organization-enforced rotation or expiry. The proposed system uses GitHub Actions and a GitHub App to generate repository secrets that rotate automatically while preserving PAT-like references in existing workflows. Its serverless design delegates scheduling, managed infrastructure, approvals, secret management, auditing, and break-glass operation to GitHub, while a custom TypeScript action encrypts and writes generated secrets. Prototyping exposed that parallel one-second jobs could accrue multiple billable minutes, prompting sequential execution in a consolidated workflow; the stated goal is improved resilience to developer churn and reduced leaked-secret blast radius.


### [What's new in Postgres 15?](https://yomu.fyi/post/what-s-new-in-postgres-15.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: pavel
- Published: Dec 16, 2022
- Type: Explainer

PostgreSQL 15, released as a stable version in October 2022, adds security, performance, SQL, replication, logging, compression, and string-processing changes. CREATE permission is revoked from all users except the database owner, views can use caller permissions with security\_invoker=on, and sorting is reported up to four times faster for in-memory and on-disk workloads; window functions also improve. The release introduces MERGE, supports two-phase commits and selective data in logical replication, allows conflicting transactions to be skipped or subscriptions disabled, and enables parallel workers for SELECT DISTINCT. It also adds JSON logging, Zstandard and ZL4 compression for WAL and pg\_basebackup, and POSIX regexp functions such as regexp\_substr() and regexp\_count(). The post recommends upgrading for bug fixes and stability, while noting that migrated databases may need manual permission changes and upgrades should be checked for rare manual actions.


### [Point in Time Recovery is now available for Pro projects](https://yomu.fyi/post/point-in-time-recovery-is-now-available-for-pro-projects.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: angelico\_de\_los\_reyes
- Published: Dec 16, 2022
- Type: Announcement

Point-in-Time Recovery (PITR) is now available to eligible Pro Plan projects, expanding access beyond Enterprise customers and adding a Dashboard UI for enabling it. Projects created from August 2022 onward can enable PITR except in Northeast Asia / Seoul, and they must use the Small compute add-on; older projects are not yet generally eligible. The system combines daily physical database snapshots with archived Write Ahead Log (WAL) records, using the open-source WAL-G tool; snapshots are uploaded daily and WAL files are archived as transactions occur. WAL files are backed up at two-minute intervals, potentially faster when a file threshold is reached, reducing the recovery point from the up-to-24-hour exposure of daily backups. PITR backups retain up to seven days by default, with self-serve expansion to 28 days, and restoration can target a specified point in time.


### [Planning in Bets: Risk Mitigation at Scale](https://yomu.fyi/post/planning-in-bets-risk-mitigation-at-scale.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Dec 6, 2022
- Type: Explainer

Preparing for BFCM and other high-traffic events, Shopify describes risk mitigation for a globally distributed, complex, interconnected infrastructure platform that peaked at 75.98M requests per minute, or 1.27M per second. At this scale, identifying and mitigating every possible failure is impossible, so the process starts with “what could go wrong” exercises, then uses voting, expert review of likelihood and severity, and assigned ownership to prioritize action. Decision makers are empowered to weigh risks and rewards, while summarized findings, including key risks and single points of failure, are shared to maintain alignment and awareness. The post frames engineering choices as probabilistic bets: decision quality should be judged separately from outcomes, and uncertainty cannot be eliminated, only addressed through thoughtful prioritization and communication.


### [Caching Without Marshal Part 1: Marshal from the Inside Out](https://yomu.fyi/post/caching-without-marshal-part-1-marshal-from-the-inside-out.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Nov 11, 2022
- Type: Explainer

Rails caching commonly relies on Ruby’s Marshal serialization, which can encode complex objects but also embeds their class names in cache entries. Shopify describes an incident in which a beta-flag refactor changed classes while old and new code overlapped during deployment; old code then failed on cached instances containing unfamiliar class names and methods, despite passing CI. The article examines Marshal through Ruby’s marshal.c implementation, covering atomic and composite types, instance variables, object references, circularity via TYPE\_LINK, and core-type subclasses via TYPE\_UCLASS. It presents MessagePack as a more compact, stricter, and controllable alternative intended to make caching safer by default, while deferring the cache migration and implementation details to the series’ next part.


### [The Management Poles of Developer Infrastructure Teams](https://yomu.fyi/post/the-management-poles-of-developer-infrastructure-teams.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Oct 28, 2022
- Type: Explainer

Developer infrastructure managers must balance three poles: management support, system and domain expertise, and organization-wide road maps, with the tension intensified by teams that often lack dedicated product managers and support critical systems for hundreds or thousands of users with six to eight developers. Domain-focused teams build deep knowledge needed for maintenance, migrations, incident investigation, on-call coverage, and developer-experience improvements, but their local road maps can diverge from the department’s highest-impact opportunities. The proposed responses are temporary individual assignments, having a whole team contribute to another team’s goals, and forming a cross-functional tiger team; each shifts trade-offs among focus, maintenance, expertise, motivation, and management oversight. A month-long tiger team built the first Spin proof of concept, which led to a dedicated team, but no structure eliminates the tensions, so choices must be judged according to context.


### [Hubble: Our Tool for Encapsulating and Extending Security Tools](https://yomu.fyi/post/hubble-our-tool-for-encapsulating-and-extending-security-tools.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Oct 21, 2022
- Type: Explainer

Shopify’s Trust team faced rising complexity in securing a distributed device fleet, including many tools, provisioning and patching work, varied networks, and the loss of in-person remediation. Hubble was built as a unified layer over mobile device management and other security systems, ingesting and standardizing their data while sending commands back with granular access controls and centralized auditing. IT staff use it for inventory, device management, and security, while employees can view device health and compliance, receive remediation guidance, and manage test-device or beta participation. The platform also serves as a standardized source for automation, and the post presents encapsulation and cross-team investment as ways to reduce administrative overhead, support proactive security, and limit risk.


[Newer posts](https://yomu.fyi/topic/reliability/page/10.md) · [Older posts](https://yomu.fyi/topic/reliability/page/12.md)