# Testing > 42 posts about Testing, summarised, each linking to the original. ## Articles ### [How we improved our iOS CI infrastructure with observability tools](https://yomu.fyi/post/how-we-improved-our-ios-ci-infrastructure-with-observability-tools.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Bunty Madan - Published: May 18, 2023 Upgrading to Xcode 13.1 introduced severe CI test instability and high CPU utilisation for Grab's iOS development team. To address this, the team integrated observability tools across their UITest pipeline to pinpoint performance bottlenecks and test flakiness. Interventions included isolating spotlight.app to curb CPU spikes, replacing Safari with a mock browser for deep link tests, and booting simulators with pre-granted permissions. The team also built custom network tracking tools to enforce resource mocking and transitioned tests away from arbitrary sleep commands to explicit wait wrappers. These combined changes cut CI runtimes, decreased CPU utilisation by over 50%, and stabilized automated test executions. ### [Evolution of quality at Grab](https://yomu.fyi/post/evolution-of-quality-at-grab.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Abby Alcantara - Published: Mar 31, 2023 As Grab expanded its superapp, manual QA sign-offs led to prolonged testing cycles and a peak in major production defects around 2019. To counter defect leakage and improve delivery pace, the engineering organization transitioned to a shift-left testing strategy across its software development lifecycle. Quality engineers began writing Given/When/Then acceptance tests prior to coding so that software engineers could execute them during development, accompanied by Definition of Ready and Definition of Done standards. The team also structured a multi-layer test strategy spanning unit tests, pipeline-executed UI component tests with mocked APIs, backend integration tests, and a custom internal end-to-end framework. Between 2019 and 2022, these changes reduced major and critical production issues by 60% and development-phase critical bugs by 40%. ### [Reflecting on the Five Years of Bug Bounty at Grab](https://yomu.fyi/post/reflecting-on-the-five-years-of-bug-bounty-at-grab.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Ajay Srivastava - Published: Dec 16, 2020 Grab launched a private bug bounty programme on HackerOne before opening it publicly in 2017 to complement internal product security efforts. Prior to going public, the team conducted security sweeps, established policies, and expanded the testing scope to prepare for an influx of reports. Operationally, the team opted to pay full bounties immediately upon report triage rather than waiting for vulnerability resolution to incentivize security researchers. Noise reduction was handled using HackerOne Triage, Human-Augmented Signal, and by blocking automated scanning networks targeting Grab's infrastructure. In addition, rotating security engineers weekly and integrating HackerOne APIs with PagerDuty ensured fast initial response times. ### [How Grab is Blazing Through the Superapp Bazel Migration](https://yomu.fyi/post/how-grab-is-blazing-through-the-superapp-bazel-migration.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Sergii Grechukha - Published: Dec 3, 2020 Grab's mobile superapp scaled past 2.5 million lines of code across both Android and iOS, leading to unsustainable local and CI build times under Gradle and Xcode. To address these bottlenecks, the engineering team analyzed their dependency trees and introduced an internal tool to calculate and optimize the build critical path. They also deployed a Kubernetes-autoscaled remote build system using Mainframer for Android and implemented Test Impact Analysis to run only affected tests in pre-merge validation. While dependency decoupling yielded modest 7% to 10% gains and iOS remote builds proved unscalable on Apple hardware, Android remote builds reduced local compile times by up to 50%, and targeted test execution reduced pre-merge pipeline durations by more than 30%. ### [Keeping 170 Libraries Up to Date on a Large Scale Android App](https://yomu.fyi/post/keeping-170-libraries-up-to-date-on-a-large-scale-android-app.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Lucas Nelaupe - Published: Oct 30, 2020 Grab's passenger Android superapp relies on more than 170 in-house and open-source libraries, incorporating five to ten library bumps into each weekly release. Although developers often avoided updates due to the fear of leaking defects or crashes into production, the engineering team established a risk-assessment framework based on codebase usage and the volume of incoming changes. To minimize update diffs and avoid accumulating large issues, libraries are updated incrementally on a weekly cadence matching upstream release schedules. Automated UI test cases written in Gherkin syntax execute on CI for every merge request, while high-risk bumps trigger targeted QA manual testing. This structured updating process prevents defect leakage while keeping the application compliant with evolving Google Play target API requirements. ### [Tackling UI Test Execution Time Imbalance for Xcode Parallel Testing](https://yomu.fyi/post/tackling-ui-test-execution-time-imbalance-for-xcode-parallel-testing.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Ngoc Thuyen Trinh - Published: Mar 16, 2020 Parallel test execution in Xcode can suffer from test time imbalance when tasks finish at significantly different times across parallel simulator workers. Analysis of Xcode scheduling logs shows that the runner groups tests by test class and dispatches all tests from the same class to a single simulator. Attempts to customize the suite by swizzling XCTestSuite fail because made-up suites initialize only after tests are dispatched. To overcome this grouping constraint, unique tokens or test names are appended to the class name component in \`-only-testing\` command-line arguments. This trick forces Xcode to treat each test as an independent class, successfully distributing individual tests across separate workers. ### [Marionette - Enabling E2E User-scenario Simulation](https://yomu.fyi/post/marionette-enabling-e2e-user-scenario-simulation.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Anish Jha - Published: Dec 23, 2019 Conducting end-to-end testing across Grab's transport microservices became difficult due to service availability, environment construction, cross-service authentication, and complex data setups for real-world user accounts. To address these challenges without relying on physical mobile devices or emulators, Grab built Marionette, an internal simulation platform for passenger and driver interactions. The platform provisions required test data, coordinates booking lifecycles, and isolates test executions across distinct user groups using localized cohorts. Engineers can configure driver and passenger behaviors, execute workflows, and run load or integration tests through a dedicated user interface, a Go SDK, and RESTful APIs. ### [Driving Southeast Asia Forward Through People-Focused Design](https://yomu.fyi/post/driving-southeast-asia-forward-through-people-focused-design.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Philip Madeley - Published: Nov 5, 2019 Designing digital products for Southeast Asia requires tailoring user experiences to unique regional constraints and consumer behaviors across diverse populations. Users in the region often operate low-end mobile hardware on congested networks while carefully rationing prepaid mobile data. Grab addresses these challenges by designing comprehensively for non-ideal UI stacks, implementing loading skeletons, and replacing heavy video tutorials with lightweight SVG animations to minimize bandwidth consumption. Furthermore, product teams adapt to mobile-only environments by prioritizing phone number and one-time-password registrations while avoiding legacy desktop-era iconography. Visual accessibility is validated by testing UI readability on dimmed, low-resolution screens under bright ambient sunlight. ### [Why You Should Organise an Immersion Trip for Your Next Project](https://yomu.fyi/post/why-you-should-organise-an-immersion-trip-for-your-next-project.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Sherizan Sheikh - Published: May 7, 2019 Grab Ventures relies on on-the-ground immersion trips to uncover real consumer behaviors and validate hypotheses that desktop research cannot address. Placing cross-functional teams directly into target markets reveals nuanced consumer motivations, such as Indonesian grocery shoppers prioritizing physical product freshness and price sensitivity over mere convenience. Effective immersion trips require pre-fieldwork reconnaissance by local residents, collaborative hypothesis generation across business and tech disciplines, and non-leading question design. Research teams should operate in small groups of two or three alongside experienced local translators while holding structured end-of-day debriefs to synthesize contextual observations. Within the Double Diamond framework, these field insights support the Discover phase and directly feed into design sprint workshops for problem framing. ### [Loki, a Dynamic Mock Server for HTTP/TCP Testing](https://yomu.fyi/post/loki-a-dynamic-mock-server-for-http-tcp-testing.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Thuy Nguyen - Published: Apr 10, 2019 Grab built Loki, a dynamic mock server written in Golang that simulates backend services on local developer machines and CI pipelines. Mobile app testing previously suffered from heavy dependencies on complex, brittle staging environments and interconnected services communicating over HTTP, HTTPS, and TCP. Loki handles both HTTP and TCP traffic on distinct ports while exposing a unified RESTful API to manage test expectations. It provides runtime flexibility through sandboxed JavaScript execution, configurable request sequence ordering, and an in-memory cron scheduler for TCP push messages. Adopting Loki decoupled mobile releases from staging stability, improving delivery cycles and enabling automated UI testing with Espresso and XCUITest. ### [Designing Resilient Systems Beyond Retries (Part 3): Architecture Patterns and Chaos Engineering](https://yomu.fyi/post/designing-resilient-systems-beyond-retries-part-3-architecture-pattern.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Michael Cartmell - Published: Mar 27, 2019 Building resilient systems requires architectural safeguards and proactive testing beyond basic retries and circuit breakers. Architectural patterns such as idempotency keys enable safe retries without creating inconsistent state during failures. Asynchronous responses and deferrable work isolate services from downstream dependency latency and errors, though they can conflict with the fail-fast principle. To validate system behavior under stress, chaos engineering introduces intentional failures in production to test hypotheses against a defined steady state. Selectively adopting complementary patterns reduces failure points while avoiding unnecessary architectural complexity. ### [Orchestrating Chaos Using Grab's Experimentation Platform](https://yomu.fyi/post/orchestrating-chaos-using-grab-s-experimentation-platform.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Roman Atachiants - Published: Nov 23, 2018 Grab operates hundreds of microservices where failures in non-critical components can cause outages in critical user flows if fallback mechanisms are improperly configured. To validate system resilience, Grab built Chaos ExP by layering a chaos engineering SDK and dedicated web UI on top of its existing Experimentation Platform. Integrated directly into the Grab-Kit server middleware, the framework intercepts incoming requests and evaluates whether to inject failures using local variable resolution. Supported failure primitives include latency, errors, panics, rate throttling, and resource leaks to test dependent services. Combining chaos testing with experimentation telemetry enables engineers to correlate injected infrastructure disruptions with business metric impacts. ### [Reliable and Scalable Feature Toggles and A/B Testing SDK at Grab](https://yomu.fyi/post/reliable-and-scalable-feature-toggles-and-a-b-testing-sdk-at-grab.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Roman Atachiants - Published: Nov 2, 2018 Grab previously managed experiments using custom service-level code and a toggling library that queried a shared Redis instance, creating latency risks and a single point of failure across backend microservices. To achieve reliable, sub-microsecond feature evaluations, the team designed a Go SDK that resolves rollouts and A/B tests entirely in memory without runtime network I/O. Backend services periodically poll JSON-defined configuration schemas stored in Amazon S3 through a Universal Configuration Manager. The SDK evaluates contextual attributes called facets locally and pushes decision telemetry asynchronously to an S3 and Presto data lake. This architecture allows engineering and product teams to gate deployments and run server-side experiments safely without service disruption. ### [Mockers - Overcoming Testing Challenges at Grab](https://yomu.fyi/post/mockers-overcoming-testing-challenges-at-grab.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Mayank Gupta - Published: Sep 18, 2018 Grab operates over 250 microservices communicating over HTTP and gRPC, making shared staging environments costly, ambiguous in ownership, and fragile due to inconsistent data and uncoordinated deployments. To address these testing bottlenecks, Grab created Mockers, a Go SDK and CLI tool backed by a central monorepo of mock servers for local-box and CI testing. Mockers automatically generates HTTP and gRPC mock servers from Swagger specifications and protobuf files, returning configured network responses without internal business logic. By incorporating Grab's in-house chaos SDK middleware, Mockers also enables repeatable resiliency and contract testing locally without relying on code-level mocks. While Grab still mandates integration testing on distributed staging environments with live data, Mockers enables developers to detect complex defects and contract mismatches earlier. ### [Building Grab’s Experimentation Platform](https://yomu.fyi/post/building-grab-s-experimentation-platform.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Abeesh Thomas - Published: Jul 13, 2018 Grab built its internal Experimentation Platform (ExP) to replace a manual, expensive testing process that required bespoke meetings, custom logging pipelines, and service modifications for each experiment. ExP provides a unified infrastructure featuring a centralized management UI, automated real-time data streaming to S3, and SDKs for Android, iOS, and Go. The platform leverages JSON-based experiment definitions delivered through dynamic configuration management, enabling client-side evaluation without costly network calls. It addresses marketplace network effects and inter-experiment interference through mechanisms such as geo-temporal segmentation and domain-layer models. The platform has scaled to run approximately 25 concurrent experiments while computing roughly 2,500 metrics and 50,000 experiment-metric combinations daily. ### [How Grab Experimented with Chat to Drive Down Booking Cancellations](https://yomu.fyi/post/how-grab-experimented-with-chat-to-drive-down-booking-cancellations.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Ishita Parbat - Published: Mar 1, 2018 Post-allocation ride cancellations at Grab degrade the booking experience and create costly inefficiencies for both passengers and driver-partners. Internal user research and platform data confirmed that rides involving GrabChat conversations had significantly lower cancellation rates by reducing perceived wait times. To scale this interaction without extra cost, the team tested system-generated automated messages sent at varying delay intervals, styles, tones, and localized verbiage across different cities. Faster message delivery outperformed longer delays, and tailored prompts reduced booking cancellations by up to two percentage points across tested markets. The experiment demonstrated that high-quality, directed prompts solicited quick responses and improved pick-up efficiency even when overall message volume was lower than control groups. ### [Grabbing Growth: A Growth Hacking Story](https://yomu.fyi/post/grabbing-growth-a-growth-hacking-story.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Gaurav Sachdeva - Published: Dec 8, 2017 Grab established a dedicated Growth Hacking team within its Technology organization to pursue high-risk, niche initiatives and scale impact across 68 million regional users. Positioning the team within engineering allowed rapid A/B testing and simultaneous multi-market deployments driven by structured growth loops. The team prioritizes and evaluates all initiatives using a Growth Factor metric, calculated as the increase in rides divided by the increase in costs. To improve driver engagement, the team implemented a Spin-to-Win game based on B.F. Skinner's variable ratio reinforcement principles, delivering probabilistic monetary and merchandise rewards upon meeting daily ride thresholds. Ongoing regional experiments continue to evaluate metrics including driver acceptance, cancellation rates, and driver ratings. ### [Come and #hackallthethings at Grab](https://yomu.fyi/post/come-and-hackallthethings-at-grab.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Grab Engineering - Published: Jul 11, 2017 Grab has officially launched a public bug bounty program in partnership with HackerOne to strengthen the security of its platform. This rollout follows a private bounty initiative operated over the previous year, during which the organization worked with over 350 security researchers and resolved nearly 200 awarded bug reports. The new public program invites external researchers to scrutinize Grab's code for critical flaws, including remote code execution, SQL injections, and exportable cross-site scripting vulnerabilities. To support ethical and responsible disclosure, Grab offers payouts reaching up to $10,000 per valid vulnerability report based on severity and impact. ### [Deep Dive into iOS Automation at Grab - Integration Testing](https://yomu.fyi/post/deep-dive-into-ios-automation-at-grab-integration-testing.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Sun Xiangxin - Published: Apr 18, 2017 Grab replaced third-party cloud CI services with an in-house Xcode Server pipeline to run automated iOS integration tests across an expanding engineering team. Instead of performing fresh repository clones on shared cloud instances, Xcode Server pulls code incrementally and mirrors local developer configurations. The team automated testing bots through Ruby scripts and Arcanist hooks tied to Phabricator code reviews, dynamically provisioning bots on diffs and deleting them after merges. This architecture preserves step-by-step UI test execution logs and screenshots for easier failure diagnostics. To maintain stability on on-premises hardware, pre-integration scripts clear application cache states using simctl commands. ### [This Rocket Ain't Stopping - Achieving Zero Downtime for Rails to Golang API Migration](https://yomu.fyi/post/this-rocket-ain-t-stopping-achieving-zero-downtime-for-rails-to-golang.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Lian Yuanlin - Published: Oct 18, 2016 Grab transitioned its public passenger app APIs from a legacy Rails application to a Golang service-oriented architecture to consolidate its codebase and engineering teams. Initial attempts to proxy traffic through a cloned Rails server via gRPC were abandoned after encountering TCP load imbalances during autoscaling events and memory leaks in the gRPC Ruby gem. The team pivoted to direct logic migration, porting Ruby logic directly into Go while decomposing modules into standalone services. Verification relied on log-based load testing and live shadow testing, where write operations were safely validated using mock data access layers that evaluated expected database outcomes. Production rollout progressed endpoint-by-endpoint using requests-per-second traffic throttling and prewarmed AWS Elastic Load Balancers before executing the final DNS switch. [Newer posts](https://yomu.fyi/topic/testing.md) · [Older posts](https://yomu.fyi/topic/testing/page/3.md)