---
title: "Latest reads"
description: "The engineering internet, summarised so you can actually read it."
---

# Latest reads
> The engineering internet, summarised so you can actually read it.

## Articles

### [Four Steps to Creating Effective Game Day Tests](https://yomu.fyi/post/four-steps-to-creating-effective-game-day-tests.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Nov 12, 2019
- Type: Tutorial

Game Day tests deliberately trigger failure modes in production systems and examine whether automated behavior and human response match expectations. Shopify frames the practice as preparation for outages that can arise despite code checks, reviews, and other safeguards. The method starts by inventorying possible failures, then selecting scenarios by impact and simulation difficulty rather than estimated likelihood. Each scenario becomes a controlled experiment with prerequisites, testable hypotheses, expected alerts and downstream behavior, plus recovery steps recorded in a spreadsheet. During execution, teams track results, communication, and work distribution; afterward they fix gaps, review coordination, update response guidelines, and rerun tests to verify changes, with periodic testing keeping systems and people prepared.


### [Driving Southeast Asia Forward Through People-Focused Design](https://yomu.fyi/post/driving-southeast-asia-forward-through-people-focused-design.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Philip Madeley
- Published: Nov 5, 2019
- Type: Explainer

Designing digital products for Southeast Asia requires tailoring user experiences to unique regional constraints and consumer behaviors across diverse populations. Users in the region often operate low-end mobile hardware on congested networks while carefully rationing prepaid mobile data. Grab addresses these challenges by designing comprehensively for non-ideal UI stacks, implementing loading skeletons, and replacing heavy video tutorials with lightweight SVG animations to minimize bandwidth consumption. Furthermore, product teams adapt to mobile-only environments by prioritizing phone number and one-time-password registrations while avoiding legacy desktop-era iconography. Visual accessibility is validated by testing UI readability on dimmed, low-resolution screens under bright ambient sunlight.


### [Sam Saffron AMA: Performance and Monitoring with Ruby](https://yomu.fyi/post/sam-saffron-ama-performance-and-monitoring-with-ruby.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Nov 5, 2019
- Type: Explainer

Sam Saffron discusses Discourse’s approach to Ruby performance and monitoring, arguing that production memory constraints matter more than isolated microbenchmarks for many deployments. He favors clear code and selective optimization, sometimes bypassing ActiveRecord with MiniSql on performance-sensitive paths rather than broadly sacrificing Ruby’s readability. Discourse keeps performance under continuous observation, and Saffron describes budgets for dependencies, boot time, and high-profile pages, including alerts for query-count regressions. He identifies memory leaks as especially difficult to diagnose, describes bisecting an application to isolate a V8-Ruby interop leak, and says MRI remains the only feasible runtime for Discourse while memory profiling and analysis tooling remain substantially behind Java and .NET.


### [Static Typing in Ruby with a Side of Sorbet](https://yomu.fyi/post/static-typing-in-ruby-with-a-side-of-sorbet.md)
- Company: [Heroku](https://yomu.fyi/company/heroku.md)
- Author: Jason Draper
- Published: Oct 31, 2019
- Type: Explainer

An experiment introduced Sorbet into a greenfield Ruby codebase with four developers to test whether static typing could catch bugs earlier, ease refactoring, and improve code design. Sorbet combines static linting with runtime checks that verify method parameters and return values against signatures, while RBI files describe methods from gems and Rails models. Setup required sorbet init, running sorbet tc, and adding that command to CI; however, Rails metaprogramming required model signatures, and sorbet-rails could miss or mistype fields. Sorbet immediately exposed issues including repeated method calls, unintended nullable database columns, and unnecessary explicit returns or guard clauses. The team reports a very positive short-term experience, while noting that the project’s young age caused most of the bugs encountered and that long-term value remains uncertain.


### [Griffin, an Anti-fraud Risk Rule Engine Making Billions of Predictions Daily](https://yomu.fyi/post/griffin-an-anti-fraud-risk-rule-engine-making-billions-of-predictions.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Muqi Li
- Published: Oct 28, 2019
- Type: Problem & solution

Grab's Trust/Identity/Safety team built Griffin, an in-house anti-fraud risk rule engine designed to process billions of daily predictions across multiple business verticals. Initially, Grab managed fraud rules directly within backend service code, but escalating rule complexity, tight rule interdependencies, and translation gaps between data scientists and developers caused deployment delays and misfiring errors. To overcome the limitations and steep learning curves of third-party engines like Drools, the team separated the workflow into data orchestration and rule-based prediction. Griffin enables analysts and data scientists to author Python-based rules directly via a web portal and reload updated logic into memory without manual developer intervention. By eliminating I/O during rule evaluation and leveraging Gunicorn multi-processing, the engine handles over 100,000 queries per second at peak on six EC2 instances with single-prediction latencies under six milliseconds.


### [Automated Continuous Deployment at Heroku](https://yomu.fyi/post/automated-continuous-deployment-at-heroku.md)
- Company: [Heroku](https://yomu.fyi/company/heroku.md)
- Author: Bernerd Schaefer
- Published: Oct 22, 2019
- Type: Problem & solution

Heroku’s Runtime team replaced a manual, coordination-heavy deployment process after team growth and Common Runtime sharding expanded work from one infrequently changed application in two environments to multiple services across 12 environments. Existing Heroku Pipelines deploy merged, CI-passing changes to staging, while the Go-based cedar-service-deployer reconciles differences between pipeline stages and promotes them when a series of safety checks pass. Those checks include a 30-minute age gate, deployment and incident controls, master-branch verification, Librato alerts, and HTTP 200 checks for configured URLs, including platform integration tests. The stateless system can handle concurrent rollouts, and the team added practices such as isolated early stages, feature flags, and separated database-migration changes. After about two years, it managed 23 services, allowing engineers to merge and rely on healthy systems for global release.


### [Make Great Decisions Quickly with TOMASP](https://yomu.fyi/post/make-great-decisions-quickly-with-tomasp.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Oct 22, 2019
- Type: Tutorial

Technical leaders and managers face high-impact decisions about hiring, technology, architecture, and priorities, but cognitive biases, short-term emotion, and overconfidence can distort judgment. TOMASP is a framework for making decisions timely, considering alternatives, defining decision criteria, examining uncertainty, and preparing to be wrong. It recommends timeboxing the decision, generating at least three options, making a meta-decision about optimization criteria, analyzing options, stepping back from emotion, and preparing to be wrong through prototypes and tripwires. In the Flutter rewrite example, Michelle compares alternatives against cross-platform consistency, performance, and development speed, then sets a two-week prototype and blind test. The framework treats decision quality as a function of how a choice is made and implemented, not simply its eventual outcomes.


### [Using Grab’s Trust Counter Service to Detect Fraud Successfully](https://yomu.fyi/post/using-grab-s-trust-counter-service-to-detect-fraud-successfully.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Chao Wang
- Published: Oct 21, 2019
- Type: Problem & solution

Grab's Trust Platform team built the Counter service to detect fraud across business verticals like transportation, food, and payments. The platform replaces manual, multi-week engineering cycles with a self-service UI where data analysts can define and experiment with counters independently. Operating on an asynchronous ingestion and synchronous transaction model, the architecture evaluates incoming stream data, enriches it via internal services, and persists aggregated signals to ScyllaDB through Grab-Stats. A multi-bucket strategy partitions queries into fifteen-minute, hourly, and daily granularities to maintain low-latency query aggregations across wide time ranges under strict SLAs.


### [Five Common Data Stores and When to Use Them](https://yomu.fyi/post/five-common-data-stores-and-when-to-use-them.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Oct 15, 2019
- Type: Explainer

The article compares five common data stores—relational databases, NoSQL databases, key-value stores, full-text search engines, and message queues—and connects each to appropriate design needs. Relational databases use schema-bound tables, primary and foreign keys, SQL, and ACID transactions for durable business-critical data, while NoSQL databases trade strict schemas for flexibility with document and wide-column models. In-memory stores such as Redis and Memcached provide fast access and configurable eviction for caching; Redis additionally supports richer data structures and disk persistence. Full-text search engines target word and substring searches, whereas Kafka uses a distributed append-only log and producer-consumer model to move data reliably between services. The article concludes that the right choice depends on whether data must be permanent, flexible, quickly accessed, searched, or transferred.


### [How to Write Fast Code in Ruby on Rails](https://yomu.fyi/post/how-to-write-fast-code-in-ruby-on-rails.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Oct 8, 2019
- Type: Tutorial

Shopify’s guide presents performance advice for Ruby on Rails across Active Record, Rails, and Ruby, while treating speed as a feature rather than the first optimization priority. It recommends understanding Active Record’s lazy query execution, selecting fewer columns, avoiding unindexed queries, using safe indexing approaches for large tables, and treating query cache as short-lived rather than dependable. For Rails applications, it covers caching, throttling expensive or abusive operations, moving long-running work into Active Job-backed queues, and reducing dependency growth to limit boot time and memory use. Ruby-specific guidance includes limiting metaprogramming and indirection, choosing O(1) hash lookups over O(n) array searches when appropriate, and reducing allocations while avoiding harmful global mutation; benchmark figures show method-definition and invocation costs can differ.


### [How Shopify Manages Petabyte Scale MySQL Backup and Restore](https://yomu.fyi/post/how-shopify-manages-petabyte-scale-mysql-backup-and-restore.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Oct 1, 2019
- Type: Problem & solution

Shopify manages a petabyte-scale fleet of MySQL servers across replica-sets, or shards, in three Google Cloud Platform regions, where file-based Xtrabackup archives made shard restores take more than six hours. To reduce that Recovery Time Objective, the team built Kubernetes CronJobs around the Compute Engine Persistent Disk snapshot API, scheduling snapshots every 15 minutes while accounting for regions, zones, instance roles, and MySQL consistency variables. Persistent Disk snapshots take about 20 minutes initially and typically under 10 minutes incrementally; restoring the latest snapshot to a new volume and starting MySQL, including InnoDB instance recovery and replication-lag recovery, typically brings RTO below 30 minutes. Retention tooling keeps the latest two snapshots plus policy-defined dailies and weeklies, while verification checks startup, GTID auto-positioning, and InnoDB page corruption; offsite tooling compresses, encrypts, and transfers exported data to another provider.


### [How Shopify Scales Up Its Development Teams](https://yomu.fyi/post/how-shopify-scales-up-its-development-teams.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Sep 26, 2019
- Type: Explainer

The article argues that scaling development is primarily about creating more effective teams, not simply increasing headcount. It first recommends checking whether an existing team prioritizes impactful work, minimizes supporting activity, and can ship code quickly while finding bugs; otherwise hiring may be premature. Shopify organizes product work into problem spaces and teams of five to nine, uses a multi-step process combining technical exercises with cultural assessment, and aims for at least as many onboarded members as onboarding members. New teams receive a baseline for work practices and technology, while transparent technical design, shared tooling with dashboards and embedded quality and testing standards, optional pairing, and frequent retrospectives support consistency and learning. The conclusion is that these practices, rather than hiring alone, enable development teams to scale more effectively.


### [Being a Principal Engineer at Grab](https://yomu.fyi/post/being-a-principal-engineer-at-grab.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Roman Atachiants
- Published: Sep 25, 2019
- Type: Explainer

Grab's rapid growth resulted in roughly 350 microservices powering its superapp, creating the need for defined individual contributor career milestones. At Grab, a principal engineer oversees the architecture of an entire Tech Family, a sub-organisation containing over 50 engineers and 20 or more microservices. Responsibilities include translating broad, ambiguous problems into concrete projects, managing technical debt, and aligning multiple engineering teams across global R&D centres. The role demands continuous technical leadership through RFC design reviews, cross-functional communication, mentorship, and self-directed prioritization alongside engineering leadership. Ultimately, principal engineers amplify engineering quality and operational stability without directly managing people.


### [Want to Improve UI Performance? Start by Understanding Your User](https://yomu.fyi/post/want-to-improve-ui-performance-start-by-understanding-your-user.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Sep 20, 2019
- Type: Tutorial

Shopify’s team examined UI performance in the Marketing section of the Shopify admin, aiming to improve load time, perceived load time, and interactions after navigation. They tested the experience on a low-end Moto G3, profiled browser activity with Chrome Developer Tools and React DevTools, and used merchant workflows to reorder priorities. The work reduced unnecessary initial work by memoizing repeated date-formatting tasks, deferring the nonessential ImagePicker bundle, and prefetching that component and its data when a merchant hovered over its activator; memoization cut one component’s render time from about 64.7 ms to 0.5 ms. For loading states, the team recommends stable skeletons with static headings and immediately useful controls, such as the Create campaign button, so merchants can act before unrelated data arrives.


### [Up to 75% Faster Maintenances with Heroku Postgres and Redis Premium Plans](https://yomu.fyi/post/up-to-75-faster-maintenances-with-heroku-postgres-and-redis-premium-pl.md)
- Company: [Heroku](https://yomu.fyi/company/heroku.md)
- Author: Corey Purcell
- Published: Aug 28, 2019
- Type: Explainer

Heroku describes how it minimizes downtime during routine security and patching maintenances for high-availability Heroku Postgres and Heroku Redis plans. These plans run a writable primary alongside a hidden read-only standby, and maintenance requires moving application writes safely to the standby before promoting it. Previously, Heroku changed DATABASE\_URL and restarted app dynos, then waited for the old primary to shut down and replicate remaining writes, which could extend downtime during long-running transactions. The new method redirects new connections to the new primary, kills existing connections to the old primary, and relies on reconnections plus a rolling restart to prevent further writes there. Heroku reports HA maintenances now complete in 15–40 seconds instead of 60 seconds or more, and says the behavior is default for Premium, Private, and Shield plans.


### [Building Resilient GraphQL APIs Using Idempotency](https://yomu.fyi/post/building-resilient-graphql-apis-using-idempotency.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Aug 27, 2019
- Type: Problem & solution

Shopify’s Payment Service uses API-level idempotency to make GraphQL payment mutations resilient to timeouts, connection failures, and ambiguous outcomes that could otherwise leave merchants unpaid or double-charge buyers. The design places a UUID-based idempotency key directly in each mutation input, identifies requests by client plus key, and locks concurrent duplicates so retries receive a 409 response while the original call is processing. An IncomingRequest record stores request state and completed recovery points, allowing completed responses to be replayed and incomplete attempts to recover before continuing. Handlers divide work into no-side-effect, transactional local-side-effect, and remote-side-effect steps, each optionally providing run and recover functions. The approach adds database-write overhead and requires careful compatibility testing when recovery points change, but the team reports that its structured handlers are clear and maintainable.


### [Designing for Accessibility: Contrast Ratio](https://yomu.fyi/post/designing-for-accessibility-contrast-ratio.md)
- Company: [Heroku](https://yomu.fyi/company/heroku.md)
- Author: Ariana Escobar
- Published: Aug 21, 2019
- Type: Problem & solution

Heroku’s accessibility initiative targeted WCAG 2.0 Level AA luminance contrast for its user interfaces, addressing reduced contrast sensitivity and color-vision deficiencies. The team audited its palette with Contrast Grid, proposed darker and revised colors, then chose a narrower proposal that preserved existing names and colors while meeting accessibility concerns within an affordable implementation effort. Because the palette affected GUIs built with Purple 3 and Malibu, pull requests generated Review Apps so teams could audit production-ready CSS and SVG assets across Dashboard components before release. Audits using axe-core 3.2.2 covered common and unusual elements, reducing color-contrast violations by an order of magnitude before rollout. The updated palette now spans the Heroku Dashboard, with planned accessibility testing through Ember tooling and further audits across other interfaces.


### [Living on the Edge of Rails](https://yomu.fyi/post/living-on-the-edge-of-rails.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Aug 21, 2019
- Type: Explainer

Shopify describes running its Core monolith on the unreleased Rails 6 master branch, or “Edge of Rails,” since February 2019, with the goal of continuously using the framework’s latest revision. Instead of annual upgrades, weekly changes reduce diff size and let developers use upstream features quickly, while a 130,000-plus-test suite finds edge cases and informs patches to Rails and related projects such as Zeitwerk. Solid Track automates weekly Rails bumps, opens GitHub pull requests, and triggers git bisect when CI fails to identify the breaking commit. Green builds move through ten-minute canary deployments before reaching the remaining servers. The approach trades the stability of released dependencies for faster fixes and features, relying on monitoring, tests, and controlled deployment to manage risk.


### [Pagination with Relative Cursors](https://yomu.fyi/post/pagination-with-relative-cursors.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Aug 12, 2019
- Type: Explainer

Incremental page-number pagination becomes increasingly expensive for large offsets, because the database scans and discards earlier records before returning a page. Shopify describes relative cursor pagination, which remembers the last record seen and continues from that point, using an indexed id to keep query time nearly constant as pages increase. In tests on a shop with 14 million products, a 100,000 offset took 2,221.60 ms, compared with 5.24 ms using the last id, while offsets of 1,000,000 and above consistently timed out. Sorting by another field requires retaining both its last value and a unique secondary key such as id, with a matching compound index to avoid skipped records and maintain performance. Shopify introduced Link-header URLs for next and previous pages and began migrating REST endpoints from page numbers, reporting early /admin/products.json usage as about 11 times faster, though adoption remained low and full results were pending.


### [Data First, SLA Always](https://yomu.fyi/post/data-first-sla-always.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Johan Kok
- Published: Aug 1, 2019
- Type: Problem & solution

Grab's Data Engineering team transitioned from periodic batch ETL ingestion to a real-time change data capture architecture called Trailblazer after dataset sizes exceeded the petabyte mark. The previous system caused severe JDBC timeouts and heavy CPU loads when executing chunked or full-scan queries on unindexed upstream MySQL tables. To solve this, MySQL binary logs are captured via Debezium on Kafka Connect, buffered in Kafka, and ingested into a data lake using Spark Structured Streaming. Checkpoints are decoupled from local storage and persisted in a Redis cluster to simplify ingestion offset overrides and handle ephemeral compute clusters. The system incorporates extensive health monitoring across Airflow, Datadog, and custom services to maintain stream liveliness and avoid Kafka retention breaches.


[Newer posts](https://yomu.fyi/posts/page/236.md) · [Older posts](https://yomu.fyi/posts/page/238.md)