---
title: "Testing"
description: "271 posts about Testing, summarised, each linking to the original."
---

# Testing
> 271 posts about Testing, summarised, each linking to the original.

## Articles

### [Resiliency Planning for High-Traffic Events](https://yomu.fyi/post/resiliency-planning-for-high-traffic-events.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Dec 10, 2020
- Type: Explainer

Shopify’s preparation for Black Friday Cyber Monday (BFCM) centers on building and maintaining resiliency practices for a high-traffic platform and large development teams. Regular load and stress tests verify required capacity, probe limits, detect regressions, and compare behavior across differently configured shops. Teams document expected user experiences and dependency failure modes in a resiliency matrix, then use game days to test that model against reality, including alerts, paging, operator readiness, and response procedures. Incident analysis treats disruptions as learning opportunities rather than stopping at a single “root cause”; nearer to BFCM, Shopify slows feature and service changes while increasing work on performance, reliability, scale, runbooks, tools, and bottlenecks. The source says these practices support a platform that powered over $5.1 billion in sales.


### [How Grab is Blazing Through the Superapp Bazel Migration](https://yomu.fyi/post/how-grab-is-blazing-through-the-superapp-bazel-migration.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Sergii Grechukha
- Published: Dec 3, 2020
- Type: Problem & solution

Grab's mobile superapp scaled past 2.5 million lines of code across both Android and iOS, leading to unsustainable local and CI build times under Gradle and Xcode. To address these bottlenecks, the engineering team analyzed their dependency trees and introduced an internal tool to calculate and optimize the build critical path. They also deployed a Kubernetes-autoscaled remote build system using Mainframer for Android and implemented Test Impact Analysis to run only affected tests in pre-merge validation. While dependency decoupling yielded modest 7% to 10% gains and iOS remote builds proved unscalable on Apple hardware, Android remote builds reduced local compile times by up to 50%, and targeted test execution reduced pre-merge pipeline durations by more than 30%.


### [Capacity Planning at Scale](https://yomu.fyi/post/capacity-planning-at-scale.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Dec 3, 2020
- Type: Explainer

Shopify’s BFCM preparation combines traffic forecasting, resource planning, and platform-wide scalability testing to reduce the risk of outages during its busiest sales period. The engineering team worked with data scientists to model expected traffic, reviewed historical numbers, added safety margins, and created a master Google Cloud resource plan covering CPUs, storage, machine types, locations, and project-level quotas. Because 2020 brought unusual uncertainty from COVID-19-driven changes in online commerce, Shopify modeled multiple demand scenarios and chose the more aggressive growth cases, while retaining buffers for changes and regional failover. It also ran the Oktoberfest scale-up on Google Kubernetes Engine, using internal load-testing tooling powered by go-lua and embedded Lua to simulate storefront browsing, cart additions, checkout, and mock payment processing. The exercise exposed bottlenecks, which teams fixed before repeating the test; the successful rerun increased confidence for BFCM and reinforced a longer-term aim of more automated, dynamic, and resilient systems.


### [Pummelling the Platform–Performance Testing Shopify](https://yomu.fyi/post/pummelling-the-platform-performance-testing-shopify.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Dec 2, 2020
- Type: Explainer

Shopify prepares for high-traffic sales events such as BFCM by repeatedly simulating customer traffic on representative test shops, aiming to expose bottlenecks before merchants experience them. Its Platform Conditioning team supports service-owning teams with approachable performance-testing tools and processes, distinguishing load tests for known traffic levels from stress tests that push services toward failure. A Go load generator runs Lua flows through lightweight VMs, models realistic traffic shapes and costly endpoints, and can issue tens of millions of requests per minute or more. Tests are initiated through Slack using the internal ChatOps tool Spy, which returns links to dashboards, logs, and results. Cronograma provides formal experiments with hypotheses, observations, conclusions, alert detection, Slack timelines, and comparable metrics such as response times, 5xx errors, and requests per minute; the tools have led to performance improvements and give Shopify confidence in future sales events.


### [How to Build a Production Grade Workflow with SQL Modelling](https://yomu.fyi/post/how-to-build-a-production-grade-workflow-with-sql-modelling.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Nov 19, 2020
- Type: Problem & solution

Shopify’s Starscream data platform handled 76,000 jobs and wrote 300 terabytes daily, but its PySpark workflow was slow and cumbersome for commonplace reporting. An audit found that about 70 percent of PySpark jobs were full batch queries that did not require generalized computing, while users often wrote SQL before translating it into Python. The team built Seamster with dbt and Google BigQuery, adding source and model structures, ownership metadata, Python-based unit tests with fixed mock inputs, CI validation, and warehouse rules. A base layer isolates raw-source changes, while models are organized into base, application-ready, and presentation layers and defined through SQL, schema, README, and test files. After a multiweek beta, most users reported shipping models in days rather than weeks, although dbt’s incremental support could not safely handle some late-arriving-data, key-resolution, rebuild, and very large-model cases.


### [Keeping 170 Libraries Up to Date on a Large Scale Android App](https://yomu.fyi/post/keeping-170-libraries-up-to-date-on-a-large-scale-android-app.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Lucas Nelaupe
- Published: Oct 30, 2020
- Type: Problem & solution

Grab's passenger Android superapp relies on more than 170 in-house and open-source libraries, incorporating five to ten library bumps into each weekly release. Although developers often avoided updates due to the fear of leaking defects or crashes into production, the engineering team established a risk-assessment framework based on codebase usage and the volume of incoming changes. To minimize update diffs and avoid accumulating large issues, libraries are updated incrementally on a weekly cadence matching upstream release schedules. Automated UI test cases written in Gherkin syntax execute on CI for every merge request, while high-risk bumps trigger targeted QA manual testing. This structured updating process prevents defect leakage while keeping the application compliant with evolving Google Play target API requirements.


### [How to Build an Experiment Pipeline from Scratch](https://yomu.fyi/post/how-to-build-an-experiment-pipeline-from-scratch.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Oct 15, 2020
- Type: Tutorial

The post describes building an email experimentation pipeline at Shopify after an existing framework for web properties and Shopify admin experiences could not support tests in external channels. The motivating case was a personalized blog-post recommendation algorithm whose incremental benefit needed comparison with generic blog emails, while local randomization and text-file storage created risks around unsubscribed users, shop-email relationships, and overlapping audiences. After requirements gathering, system mapping, mock-table queries, stakeholder review, and technical design review, the author implemented three pipeline phases in PySpark: experiment definitions, consolidation, and filtering with deduplication and exclusion rules. Production output supported ongoing randomization, backfills, category tags, sample limits, and integration with Shopify’s existing analysis framework; the tool was subsequently used across multiple data teams for email experiments.


### [How to Use Quasi-experiments and Counterfactuals to Build Great Products](https://yomu.fyi/post/how-to-use-quasi-experiments-and-counterfactuals-to-build-great-produc.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Sep 28, 2020
- Type: Explainer

Correlation does not establish causation, so the post presents causal inference as a way to determine why product metrics change and to validate or invalidate business strategies. The proposed evidence ladder places randomized A/B tests first, quasi-experiments such as fixed-effects regression and difference-in-differences second, and counterfactual estimation third, with descriptive statistics providing no direct causal evidence. It explains that quasi-experiments use non-random treatment and control divisions, while difference-in-differences depends on a parallel-trends assumption and fixed-effects regression assumes relevant dividing factors were collected. For a security update released to everyone, the team trained a time-series model on unaffected feature usage and global activity trends, then found no lift versus actual usage, indicating no negative effect on usage. Because observational methods bring greater uncertainty and frequent false positives, the post recommends stating assumptions, relaxing them through robustness checks, and using DAGs and Dagitty to test causal hypotheses.


### [Building Reliable Mobile Applications](https://yomu.fyi/post/building-reliable-mobile-applications.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Aug 14, 2020
- Type: Explainer

Shopify’s Point Of Sale (POS) app is mission-critical for brick-and-mortar merchants, because downtime can create long lineups, unhappy customers, and lost sales. Unlike server-side systems, mobile fixes depend on Apple and Google review, and merchants may delay installing approved updates, so the team focuses on preventing bugs and shortening recovery time. Its practices include layered automated and manual testing, component ownership with automatic Code Owners reviews, automated mobile “tophatting,” and weekly releases supported by a rotating Release Captain and ShipIt Mobile. Staged rollouts, beta flags, minimum-version enforcement for rare critical security fixes, and dedicated Primary and Secondary ATCs limit exposure and clarify incident response. ChatOps-based incident channels capture findings for root cause analysis, while rotating on-call responsibilities reduce context switching; these measures helped the app scale to larger merchants while shipping faster and improving reliability.


### [How We Added Client Rate Throttling to the Platform API Gem](https://yomu.fyi/post/how-we-added-client-rate-throttling-to-the-platform-api-gem.md)
- Company: [Heroku](https://yomu.fyi/company/heroku.md)
- Author: Richard Schneeman
- Published: Jul 8, 2020
- Type: Explainer

The platform-api gem needed client-side rate throttling for Heroku’s API, which uses GCRA and allows a bucket of up to 4,500 request tokens, replenished at roughly 75 per minute. The goal was to avoid 429 responses without creating retry traffic, while operating within the client library and without coordination across machines. The work combined tests with a simulated server, processes and threads, charting, and metrics for average retry rate, maximum sleep, request-count variance, and workload-clearing time. The winning strategy adjusted sleep downward using the remaining request count relative to the 4,500-request maximum; in simulation it achieved a 3.07% retry rate, 17.32-second maximum sleep, 78.44 standard-deviation request count, and 84.23 seconds to clear 4,500 requests. The change shipped behind a null strategy before becoming the default in version 3.0.0+, with feedback reporting no significant breaks or severe application dysfunction.


### [Spark Joy by Running Fewer Tests](https://yomu.fyi/post/spark-joy-by-running-fewer-tests.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Jun 11, 2020
- Type: Problem & solution

Shopify’s monolithic Ruby repository has more than 150,000 tests, grows 20–30% annually, and takes 30–40 minutes to run across hundreds of Docker containers, making required full-suite CI costly and vulnerable to intermittent failures. The team built a dynamic-analysis test-selection system that logs method calls during each test, records files in each call graph, and maps changed files to relevant tests. Because Ruby, Rails metaprogramming, and non-Ruby files complicate tracing, the system adds Rails patches and fallback rules, runs extra tests when mappings lag, and executes the full suite asynchronously on every deploy. After two months and 8,360 merged commits, it achieved 99.94% failure recall, selected about 60% of tests, and reduced compute time by roughly 25%. Developers requested full-suite runs on fewer than 2% of pull requests.


### [Software Release Culture at Shopify](https://yomu.fyi/post/software-release-culture-at-shopify.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Apr 15, 2020
- Type: Explainer

Shopify describes its release culture as a combination of beliefs and behaviors shaping how developers ship software, with trust, velocity, and safeguards as stated goals. The team measures developer experience through biannual happiness surveys, shipping Slack channels, dogfooding, internal support, and UX research that includes shadowing developers and testing prototypes with interns and new hires. Its release pipeline runs from a pull request and /shipit command through the merge queue, Master, a Canary environment receiving 5% of incoming requests, and Production after a 10-minute testing period without intervention or automated alerts. Developers own releases without managers, sign-offs, or release windows, while /shipit --emergency provides a single fast recovery path. Shopify also uses Beta Flags for gradual rollouts and measures /shipit success through survey feedback and average pull-request time-to-production.


### [Tackling UI Test Execution Time Imbalance for Xcode Parallel Testing](https://yomu.fyi/post/tackling-ui-test-execution-time-imbalance-for-xcode-parallel-testing.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Ngoc Thuyen Trinh
- Published: Mar 16, 2020
- Type: Problem & solution

Parallel test execution in Xcode can suffer from test time imbalance when tasks finish at significantly different times across parallel simulator workers. Analysis of Xcode scheduling logs shows that the runner groups tests by test class and dispatches all tests from the same class to a single simulator. Attempts to customize the suite by swizzling XCTestSuite fail because made-up suites initialize only after tests are dispatched. To overcome this grouping constraint, unique tokens or test names are appended to the class name component in \`-only-testing\` command-line arguments. This trick forces Xcode to treat each test as an independent class, successfully distributing individual tests across separate workers.


### [Zoom Developer Relations — Meet the Team](https://yomu.fyi/post/zoom-developer-relations-meet-the-team.md)
- Company: [Zoom](https://yomu.fyi/company/zoom.md)
- Author: Abraham Queen
- Published: Jan 24, 2020
- Type: Explainer

Zoom's Developer Relations team supports third-party developers, manages the App Marketplace, and maintains platform tools including APIs and Web SDKs. The team comprises technical writers, developer advocates, developer educators, security engineers, and marketplace specialists. Specific responsibilities include authoring integration documentation, reviewing early API designs, testing webhooks, scheduling security evaluations for Marketplace applications, and building sample code. Developer interactions take place across multiple channels, including the Zoom Developer Forum, Zendesk support tickets, Zoom meetings, app submission review notes, and partnership calls. Specialists also guide creators through the formal app submission lifecycle, reviewing functional workflows, privacy statements, and regional ISV opportunities across global markets like APAC.


### [Behind the Curtain: App Publishing Requests](https://yomu.fyi/post/behind-the-curtain-app-publishing-requests.md)
- Company: [Zoom](https://yomu.fyi/company/zoom.md)
- Author: Abraham Queen
- Published: Jan 22, 2020
- Type: Explainer

Zoom App Marketplace reviews App Publishing Requests to enable broad distribution of applications across all Zoom accounts. During the initial review stage, marketplace specialists evaluate submission metadata, including descriptive text, media assets, functional documentation, and requested scopes to verify that applications do not access unnecessary user information. Reviewers then perform a functional assessment by executing the developer-provided test plan using supplied test accounts and credentials. When issues occur, reviewers deliver feedback through review notes, requiring developers to resolve the problems and resubmit their app to trigger a re-review cycle. Developers can validate production readiness prior to public submission through account-level local testing or by creating a Publishable URL sharing request for external testing.


### [Marionette - Enabling E2E User-scenario Simulation](https://yomu.fyi/post/marionette-enabling-e2e-user-scenario-simulation.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Anish Jha
- Published: Dec 23, 2019
- Type: Problem & solution

Conducting end-to-end testing across Grab's transport microservices became difficult due to service availability, environment construction, cross-service authentication, and complex data setups for real-world user accounts. To address these challenges without relying on physical mobile devices or emulators, Grab built Marionette, an internal simulation platform for passenger and driver interactions. The platform provisions required test data, coordinates booking lifecycles, and isolates test executions across distinct user groups using localized cohorts. Engineers can configure driver and passenger behaviors, execute workflows, and run load or integration tests through a dedicated user interface, a Go SDK, and RESTful APIs.


### [Successfully Merging the Work of 1000+ Developers](https://yomu.fyi/post/successfully-merging-the-work-of-1000-developers.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Nov 14, 2019
- Type: Problem & solution

Shopify’s single monolith supports changes 40 times a day and roughly 400 commits to master daily, making safe trunk-based development difficult at the scale of 1,000+ developers. Its first Merge Queue limited deployment drift and batch size, but queued pull requests did not receive CI, soft conflicts could break master, and a browser extension enabled accidental direct merges. Merge Queue v2 replaces that workflow with GitHub comments and webhooks, programmatic branch protection, an emergency /shipit --emergency path, and a predictive branch where queued changes receive CI before merging. A reconciliation algorithm synchronizes that branch through GitHub’s GraphQL API, while failure-tolerance thresholds handle flaky tests and remove repeatedly failing pull requests. CI starts when requests enter the queue; batches of eight and three concurrently tested batches balance throughput, risk, and resource use, while the changes improved user experience, deployment safety, and deploy throughput.


### [Four Steps to Creating Effective Game Day Tests](https://yomu.fyi/post/four-steps-to-creating-effective-game-day-tests.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Nov 12, 2019
- Type: Tutorial

Game Day tests deliberately trigger failure modes in production systems and examine whether automated behavior and human response match expectations. Shopify frames the practice as preparation for outages that can arise despite code checks, reviews, and other safeguards. The method starts by inventorying possible failures, then selecting scenarios by impact and simulation difficulty rather than estimated likelihood. Each scenario becomes a controlled experiment with prerequisites, testable hypotheses, expected alerts and downstream behavior, plus recovery steps recorded in a spreadsheet. During execution, teams track results, communication, and work distribution; afterward they fix gaps, review coordination, update response guidelines, and rerun tests to verify changes, with periodic testing keeping systems and people prepared.


### [Driving Southeast Asia Forward Through People-Focused Design](https://yomu.fyi/post/driving-southeast-asia-forward-through-people-focused-design.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Philip Madeley
- Published: Nov 5, 2019
- Type: Explainer

Designing digital products for Southeast Asia requires tailoring user experiences to unique regional constraints and consumer behaviors across diverse populations. Users in the region often operate low-end mobile hardware on congested networks while carefully rationing prepaid mobile data. Grab addresses these challenges by designing comprehensively for non-ideal UI stacks, implementing loading skeletons, and replacing heavy video tutorials with lightweight SVG animations to minimize bandwidth consumption. Furthermore, product teams adapt to mobile-only environments by prioritizing phone number and one-time-password registrations while avoiding legacy desktop-era iconography. Visual accessibility is validated by testing UI readability on dimmed, low-resolution screens under bright ambient sunlight.


### [How Shopify Scales Up Its Development Teams](https://yomu.fyi/post/how-shopify-scales-up-its-development-teams.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Sep 26, 2019
- Type: Explainer

The article argues that scaling development is primarily about creating more effective teams, not simply increasing headcount. It first recommends checking whether an existing team prioritizes impactful work, minimizes supporting activity, and can ship code quickly while finding bugs; otherwise hiring may be premature. Shopify organizes product work into problem spaces and teams of five to nine, uses a multi-step process combining technical exercises with cultural assessment, and aims for at least as many onboarded members as onboarding members. New teams receive a baseline for work practices and technology, while transparent technical design, shared tooling with dashboards and embedded quality and testing standards, optional pairing, and frequent retrospectives support consistency and learning. The conclusion is that these practices, rather than hiring alone, enable development teams to scale more effectively.


[Newer posts](https://yomu.fyi/topic/testing/page/11.md) · [Older posts](https://yomu.fyi/topic/testing/page/13.md)