---
title: "Testing"
description: "271 posts about Testing, summarised, each linking to the original."
---

# Testing
> 271 posts about Testing, summarised, each linking to the original.

## Articles

### [Designing for Accessibility: Contrast Ratio](https://yomu.fyi/post/designing-for-accessibility-contrast-ratio.md)
- Company: [Heroku](https://yomu.fyi/company/heroku.md)
- Author: Ariana Escobar
- Published: Aug 21, 2019
- Type: Problem & solution

Heroku’s accessibility initiative targeted WCAG 2.0 Level AA luminance contrast for its user interfaces, addressing reduced contrast sensitivity and color-vision deficiencies. The team audited its palette with Contrast Grid, proposed darker and revised colors, then chose a narrower proposal that preserved existing names and colors while meeting accessibility concerns within an affordable implementation effort. Because the palette affected GUIs built with Purple 3 and Malibu, pull requests generated Review Apps so teams could audit production-ready CSS and SVG assets across Dashboard components before release. Audits using axe-core 3.2.2 covered common and unusual elements, reducing color-contrast violations by an order of magnitude before rollout. The updated palette now spans the Heroku Dashboard, with planned accessibility testing through Ember tooling and further audits across other interfaces.


### [Living on the Edge of Rails](https://yomu.fyi/post/living-on-the-edge-of-rails.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Aug 21, 2019
- Type: Explainer

Shopify describes running its Core monolith on the unreleased Rails 6 master branch, or “Edge of Rails,” since February 2019, with the goal of continuously using the framework’s latest revision. Instead of annual upgrades, weekly changes reduce diff size and let developers use upstream features quickly, while a 130,000-plus-test suite finds edge cases and informs patches to Rails and related projects such as Zeitwerk. Solid Track automates weekly Rails bumps, opens GitHub pull requests, and triggers git bisect when CI fails to identify the breaking commit. Green builds move through ten-minute canary deployments before reaching the remaining servers. The approach trades the stability of released dependencies for faster fixes and features, relying on monitoring, tests, and controlled deployment to manage risk.


### [How Heroku Operates its Multi-Tenant Apache Kafka Services](https://yomu.fyi/post/how-heroku-operates-its-multi-tenant-apache-kafka-services.md)
- Company: [Heroku](https://yomu.fyi/company/heroku.md)
- Author: Ali Hamidi
- Published: Jul 11, 2019
- Type: Explainer

Heroku’s Apache Kafka service supports thousands of developers and millions of transactions, including customers sharing multi-tenant clusters. The post explains how those clusters provide security, predictable performance, availability, and production-like topology at lower cost than single-tenant plans. Multi-tenant and single-tenant offerings use the same numbers of brokers and ZooKeeper nodes, while Kafka ACLs, per-tenant resource namespaces, and producer/consumer byte quotas isolate customers and limit resource contention. Heroku provisions purchased resources, expands storage at 80% of quota, and uses higher defaults for partitions, replicas, and in-sync replicas; exceeding capacity can eventually throttle throughput without stopping the cluster. Automated tests model 50 users with varied traffic, exercise throughput and partition configurations, and simulate broker failures and restarts to identify operational limits and adjust the infrastructure before customer impact.


### [Componentizing Shopify’s Tax Engine](https://yomu.fyi/post/componentizing-shopify-s-tax-engine.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Jul 5, 2019
- Type: Problem & solution

Shopify’s tax engine grew from a simple system into difficult code as tax rules and ownership spread across Cart, Checkout, and Order. The team first learned the surrounding code and tax domain with help from a tax subject matter expert, then mapped dependencies using regular expressions, scripts, manual investigation, and tax-related bug work. They moved tax source files into a dedicated component behind explicit TaxesRequestSchema and TaxesReponseSchema interfaces, with simple-type request and response schemas, protected boundaries, and expanded unit, integration, and end-to-end tests. During release, the old and new engines calculated taxes concurrently so discrepancies could be measured; after iterative fixes, Shopify rolled the component out gradually from a small group of shops to all shops over several days, with no downtime or merchant impact reported.


### [Implementing Android POS Receipt Printing on Shopify](https://yomu.fyi/post/implementing-android-pos-receipt-printing-on-shopify.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Jun 5, 2019
- Type: Explainer

Shopify’s Android POS lacked printed-receipt support until March 2019, leaving merchants who needed it to use iPads. The design targets a pragmatic, adaptive, composable, maintainable, and highly testable API, while handling more than 8 billion possible receipt content combinations across discounts, tips, transactions, gift cards, refunds, payment methods, and regional requirements. Its six-step pipeline aggregates models from SQLite or GraphQL, transforms them into a PrintableReceipt, defines view-based sections, creates printer-agnostic PrintRequest objects, renders section bitmaps at the required resolution, and passes them through the Hardware SDK to vendor-specific integrations. The PrintableReceiptComposer contains most business logic, with unit tests covering features and edge cases, while screenshot tests compare rendered output pixel by pixel in CI. This componentized separation isolates printer support from receipt generation and allows new printer models to be added in a couple of hours.


### [Why You Should Organise an Immersion Trip for Your Next Project](https://yomu.fyi/post/why-you-should-organise-an-immersion-trip-for-your-next-project.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Sherizan Sheikh
- Published: May 7, 2019
- Type: Tutorial

Grab Ventures relies on on-the-ground immersion trips to uncover real consumer behaviors and validate hypotheses that desktop research cannot address. Placing cross-functional teams directly into target markets reveals nuanced consumer motivations, such as Indonesian grocery shoppers prioritizing physical product freshness and price sensitivity over mere convenience. Effective immersion trips require pre-fieldwork reconnaissance by local residents, collaborative hypothesis generation across business and tech disciplines, and non-leading question design. Research teams should operate in small groups of two or three alongside experienced local translators while holding structured end-of-day debriefs to synthesize contextual observations. Within the Double Diamond framework, these field insights support the Discover phase and directly feed into design sprint workshops for problem framing.


### [Loki, a Dynamic Mock Server for HTTP/TCP Testing](https://yomu.fyi/post/loki-a-dynamic-mock-server-for-http-tcp-testing.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Thuy Nguyen
- Published: Apr 10, 2019
- Type: Problem & solution

Grab built Loki, a dynamic mock server written in Golang that simulates backend services on local developer machines and CI pipelines. Mobile app testing previously suffered from heavy dependencies on complex, brittle staging environments and interconnected services communicating over HTTP, HTTPS, and TCP. Loki handles both HTTP and TCP traffic on distinct ports while exposing a unified RESTful API to manage test expectations. It provides runtime flexibility through sandboxed JavaScript execution, configurable request sequence ordering, and an in-memory cron scheduler for TCP push messages. Adopting Loki decoupled mobile releases from staging stability, improving delivery cycles and enabling automated UI testing with Espresso and XCUITest.


### [Designing Resilient Systems Beyond Retries (Part 3): Architecture Patterns and Chaos Engineering](https://yomu.fyi/post/designing-resilient-systems-beyond-retries-part-3-architecture-pattern.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Michael Cartmell
- Published: Mar 27, 2019
- Type: Explainer

Building resilient systems requires architectural safeguards and proactive testing beyond basic retries and circuit breakers. Architectural patterns such as idempotency keys enable safe retries without creating inconsistent state during failures. Asynchronous responses and deferrable work isolate services from downstream dependency latency and errors, though they can conflict with the fail-fast principle. To validate system behavior under stress, chaos engineering introduces intentional failures in production to test hypotheses against a defined steady state. Selectively adopting complementary patterns reduces failure points while avoiding unnecessary architectural complexity.


### [iOS Application Testing Strategies at Shopify](https://yomu.fyi/post/ios-application-testing-strategies-at-shopify.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: Uzziah Eyee
- Published: Jan 15, 2019
- Type: Explainer

Shopify describes an iOS testing strategy for a monorepo containing multiple app projects and evolving through hundreds of weekly commits. The approach covers Models, Views, ViewModels, View Controllers, and user workflows with tests matched to each component’s role: regular unit tests for business rules, snapshot tests for visual behavior, and XCUITest UI automation for user-facing flows. UI automation selects elements through accessibility labels or strings, which can also improve accessibility for visually impaired users, while manual testing remains necessary for some animation and rendering issues. The article recommends a test pyramid because slower tests require more human effort and maintenance, and says suites must evolve as business requirements change.


### [The Unreasonable Effectiveness of Test Retries: An Android Monorepo Case Study](https://yomu.fyi/post/the-unreasonable-effectiveness-of-test-retries-an-android-monorepo-cas.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: Alejandro Rodriguez Salamanca
- Published: Jan 8, 2019
- Type: Problem & solution

An Android monorepo expansion and larger team increased the number of tests run on master, exposing how flaky tests and infrastructure failures could erode CI confidence. Under an independence assumption, tests passing 99.95% each produced a theoretical full-suite pass rate of only 35% across 20 pipelines, while retrying each failed test once raised it to 99.95%. Shopify therefore added retries for setup failures, distinguishing failures retriable within a job from fatal failures requiring the test environment to reload, and retried failed tests up to three times. The system also flagged tests that passed after retry as warnings, since retries can hide reliability problems and slow builds. In practice, Android app pipeline pass rate rose from 31% to almost 90% after retry mechanisms and Gradle caching, while iOS rose from 67% to 97%.


### [Preparing Shopify for Black Friday and Cyber Monday](https://yomu.fyi/post/preparing-shopify-for-black-friday-and-cyber-monday.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: Camilo Lopez
- Published: Dec 21, 2018
- Type: Explainer

Shopify’s Production Engineering team prepares its platform and 600,000+ merchants for Black Friday and Cyber Monday traffic without interrupting service. Preparation combines controlled fault injection “game days,” synthetic load tests, and staged freezes: a feature freeze begins several weeks ahead, followed by a code freeze allowing only critical fixes during the event. The tests probe dependencies such as shipping providers and payment gateways, including latency, fallback behavior, and responses to datastore failures. After the 2018 weekend, Shopify reported nearly 11,000 orders per minute, around 100,000 requests per second for extended periods, and overall metrics about 1.8 times 2017 levels. The account concludes that fault injection, load testing, and disaster role-play improved readiness, while third-party downtime remains Shopify’s responsibility to merchants and buyers.


### [Orchestrating Chaos Using Grab's Experimentation Platform](https://yomu.fyi/post/orchestrating-chaos-using-grab-s-experimentation-platform.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Roman Atachiants
- Published: Nov 23, 2018
- Type: Problem & solution

Grab operates hundreds of microservices where failures in non-critical components can cause outages in critical user flows if fallback mechanisms are improperly configured. To validate system resilience, Grab built Chaos ExP by layering a chaos engineering SDK and dedicated web UI on top of its existing Experimentation Platform. Integrated directly into the Grab-Kit server middleware, the framework intercepts incoming requests and evaluates whether to inject failures using local variable resolution. Supported failure primitives include latency, errors, panics, rate throttling, and resource leaks to test dependent services. Combining chaos testing with experimentation telemetry enables engineers to correlate injected infrastructure disruptions with business metric impacts.


### [Reliable and Scalable Feature Toggles and A/B Testing SDK at Grab](https://yomu.fyi/post/reliable-and-scalable-feature-toggles-and-a-b-testing-sdk-at-grab.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Roman Atachiants
- Published: Nov 2, 2018
- Type: Problem & solution

Grab previously managed experiments using custom service-level code and a toggling library that queried a shared Redis instance, creating latency risks and a single point of failure across backend microservices. To achieve reliable, sub-microsecond feature evaluations, the team designed a Go SDK that resolves rollouts and A/B tests entirely in memory without runtime network I/O. Backend services periodically poll JSON-defined configuration schemas stored in Amazon S3 through a Universal Configuration Manager. The SDK evaluates contextual attributes called facets locally and pushes decision telemetry asynchronously to an S3 and Presto data lake. This architecture allows engineering and product teams to gate deployments and run server-side experiments safely without service disruption.


### [Introducing the Deprecation Toolkit](https://yomu.fyi/post/introducing-the-deprecation-toolkit.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: Edouard CHIN
- Published: Sep 19, 2018
- Type: Announcement

Shopify’s Deprecation Toolkit is an open-source Ruby gem for tracking deprecated code in an application and preventing new deprecations from entering a codebase. It addresses the limits of warning messages and the impracticality of immediately raising on every existing deprecation, especially in a large monolith with complex dependency upgrades. The toolkit records deprecations triggered by test suites in YAML whitelist files, then compares later test runs with the accepted set; mismatches trigger configurable behavior, raising an error by default. Existing deprecations can therefore remain acceptable while teams fix them gradually, whereas introduced or removed deprecations are surfaced in CI. Shopify reports that using the toolkit reduced deprecations, supported dependency upgrades, blocked pull requests introducing new ones, and enabled team-based tracking of remediation progress.


### [Mockers - Overcoming Testing Challenges at Grab](https://yomu.fyi/post/mockers-overcoming-testing-challenges-at-grab.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Mayank Gupta
- Published: Sep 18, 2018
- Type: Problem & solution

Grab operates over 250 microservices communicating over HTTP and gRPC, making shared staging environments costly, ambiguous in ownership, and fragile due to inconsistent data and uncoordinated deployments. To address these testing bottlenecks, Grab created Mockers, a Go SDK and CLI tool backed by a central monorepo of mock servers for local-box and CI testing. Mockers automatically generates HTTP and gRPC mock servers from Swagger specifications and protobuf files, returning configured network responses without internal business logic. By incorporating Grab's in-house chaos SDK middleware, Mockers also enables repeatable resiliency and contract testing locally without relying on code-level mocks. While Grab still mandates integration testing on distributed staging environments with live data, Mockers enables developers to detect complex defects and contract mismatches earlier.


### [Building Grab’s Experimentation Platform](https://yomu.fyi/post/building-grab-s-experimentation-platform.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Abeesh Thomas
- Published: Jul 13, 2018
- Type: Problem & solution

Grab built its internal Experimentation Platform (ExP) to replace a manual, expensive testing process that required bespoke meetings, custom logging pipelines, and service modifications for each experiment. ExP provides a unified infrastructure featuring a centralized management UI, automated real-time data streaming to S3, and SDKs for Android, iOS, and Go. The platform leverages JSON-based experiment definitions delivered through dynamic configuration management, enabling client-side evaluation without costly network calls. It addresses marketplace network effects and inter-experiment interference through mechanisms such as geo-temporal segmentation and domain-layer models. The platform has scaled to run approximately 25 concurrent experiments while computing roughly 2,500 metrics and 50,000 experiment-metric combinations daily.


### [Event Sourcing made Simple](https://yomu.fyi/post/event-sourcing-made-simple.md)
- Company: [Kickstarter](https://yomu.fyi/company/kickstarter.md)
- Author: Philippe Creux
- Published: Apr 20, 2018
- Type: Explainer

Event Sourcing stores application changes as immutable events, providing a history that can explain how current data came to be. The post introduces four components: Events record changes, Calculators update Aggregates from those events, and Reactors trigger side effects or create further events. Kickstarter implemented a minimal framework for d.rip using regular Active Record aggregates, a regular SQL database, synchronous atomic updates, and roughly 200 lines of home-made code. The approach supports replaying history to reconstruct past state, repair calculator bugs, backfill new aggregate fields, optimize read models, and forward events to analytics platforms, while presenting naming and mutation-to-event decomposition as important challenges.


### [How Grab Experimented with Chat to Drive Down Booking Cancellations](https://yomu.fyi/post/how-grab-experimented-with-chat-to-drive-down-booking-cancellations.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Ishita Parbat
- Published: Mar 1, 2018
- Type: Problem & solution

Post-allocation ride cancellations at Grab degrade the booking experience and create costly inefficiencies for both passengers and driver-partners. Internal user research and platform data confirmed that rides involving GrabChat conversations had significantly lower cancellation rates by reducing perceived wait times. To scale this interaction without extra cost, the team tested system-generated automated messages sent at varying delay intervals, styles, tones, and localized verbiage across different cities. Faster message delivery outperformed longer delays, and tailored prompts reduced booking cancellations by up to two percentage points across tested markets. The experiment demonstrated that high-quality, directed prompts solicited quick responses and improved pick-up efficiency even when overall message volume was lower than control groups.


### [Grabbing Growth: A Growth Hacking Story](https://yomu.fyi/post/grabbing-growth-a-growth-hacking-story.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Gaurav Sachdeva
- Published: Dec 8, 2017
- Type: Problem & solution

Grab established a dedicated Growth Hacking team within its Technology organization to pursue high-risk, niche initiatives and scale impact across 68 million regional users. Positioning the team within engineering allowed rapid A/B testing and simultaneous multi-market deployments driven by structured growth loops. The team prioritizes and evaluates all initiatives using a Growth Factor metric, calculated as the increase in rides divided by the increase in costs. To improve driver engagement, the team implemented a Spin-to-Win game based on B.F. Skinner's variable ratio reinforcement principles, delivering probabilistic monetary and merchandise rewards upon meeting daily ride thresholds. Ongoing regional experiments continue to evaluate metrics including driver acceptance, cancellation rates, and driver ratings.


### [FRP at Kickstarter and Kotlin](https://yomu.fyi/post/frp-at-kickstarter-and-kotlin.md)
- Company: [Kickstarter](https://yomu.fyi/company/kickstarter.md)
- Author: Julie
- Published: Nov 6, 2017
- Type: Tutorial

Kickstarter’s open-source iOS and Android apps use Functional Reactive Programming (FRP) patterns to keep views simple and move mutation and complex logic into thoroughly tested ViewModels. The post describes how this discipline helped a developer ramp up quickly on the Android codebase after working on Kickstarter’s backend and payments teams. It also introduces the Android team’s interest in Kotlin, a JetBrains language that runs on the JVM, while noting that dependency concerns had prevented conversion of core functionality. For a 2017 Functional Swift Conference talk in Berlin, the author used Kotlin to add a simple Android screen following the team’s established FRP patterns, presenting the implementation as live coding and an introduction to FRP and Kotlin.


[Newer posts](https://yomu.fyi/topic/testing/page/12.md) · [Older posts](https://yomu.fyi/topic/testing/page/14.md)