Loading…
Browse by type
Problem & solution reads
Problem & solution posts on Yomu, summarised newest first.
- 934 posts
- 20 companies
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Problem & solution posts
Save Your Place with Grab!
Grab introduced Saved Places across Southeast Asia to eliminate the friction of repeatedly typing addresses and prevent selection errors between similarly named locations. Data analysis across transport and food orders revealed that consumers consistently visit only five to seven unique locations and order food to one or two addresses. To streamline repeat bookings, cross-functional teams designed a feature allowing users to bookmark locations under custom labels such as Home and Work. Following usability testing and release, more than 14 million users stored nearly 45 million addresses across the platform. Platform metrics also showed that while office destinations clustered in central districts across major cities, residential distributions varied significantly between markets like Singapore and Jakarta.
How We Built a Logging Stack at Grab
Grab needed a scalable logging platform to replace slow, fragmented systems that hindered debugging across their growing service fleet. Generating 25TB of daily logs, the team built a horizontally scalable Elasticsearch cluster configured via Ansible and monitored with Datadog. Although the initial proof of concept assigned all node roles (ingest, coordinator, master, and data) to every machine, operating at scale introduced major challenges with JVM heap exhaustion and cluster stability. The team resolved memory pressure and performance bottlenecks by tuning circuit breakers, lowering field data cache limits, adjusting shard allocations based on segment memory, and disabling translog compression during shard transfers.
Componentizing Shopify’s Tax Engine
Shopify’s tax engine grew from a simple system into difficult code as tax rules and ownership spread across Cart, Checkout, and Order. The team first learned the surrounding code and tax domain with help from a tax subject matter expert, then mapped dependencies using regular expressions, scripts, manual investigation, and tax-related bug work. They moved tax source files into a dedicated component behind explicit TaxesRequestSchema and TaxesReponseSchema interfaces, with simple-type request and response schemas, protected boundaries, and expanded unit, integration, and end-to-end tests. During release, the old and new engines calculated taxes concurrently so discrepancies could be measured; after iterative fixes, Shopify rolled the component out gradually from a small group of shops to all shops over several days, with no downtime or merchant impact reported.
Catwalk: Serving Machine Learning Models at Scale
As machine learning adoption expanded at Grab, individual teams created fragmented model serving solutions that duplicated engineering effort and required data scientists to handle underlying infrastructure. To resolve these inefficiencies, Grab developed Catwalk, a self-service machine learning model serving platform. The system runs TensorFlow Serving containers across a managed Kubernetes cluster integrated with Grab's observability stack. Data scientists deploy or update models simply by saving files using the tf.saved_model API to dedicated Amazon S3 buckets, while Kubernetes automates orchestration, ingress routing, and pod autoscaling. Catwalk abstracts server management away from data scientists, shortens deployment timelines, and provides high availability during model version rollouts.
React Native in GrabPay
Following the release of the GrabPay Merchant App, Grab adopted React Native inside the Grab Passenger app to maintain a single cross-platform codebase across iOS and Android. Integrating the framework into native applications required establishing bridge communication guidelines and incorporating React Native modules into the Grablet architecture. To streamline network communication, API calls were migrated from axios to native bridges returning promises, which eliminated the need to pass access tokens into JavaScript. The team also established a shared internal library of approximately 20 UI components alongside Redux for state management and react-navigation for routing. Modules like BillPay and Transaction History successfully launched across Southeast Asia while maintaining the performance and feel of native software.
Mobile Release Engineering at Scale with Shipit Mobile
Shopify built Shipit Mobile to replace error-prone, manual mobile app releases with a shared platform for creating, viewing, and managing releases. The process creates a release branch from master, builds a candidate for each commit, increments build numbers, and starts separate CI pipelines for testing and release builds. Shipit Mobile uploads the same binary and metadata to distribution channels including Google Play, the App Store, and internal support access, using store APIs and Fastlane, while Slack notifications make release status visible beyond the release captain. Developers can bypass CI when necessary for emergencies, and configuration favors convention over configuration for project metadata. After more than six months in production, the platform reduced release complexity, shortened the release cadence from three weeks to one week, and helped more people become release captains.
Loki, a Dynamic Mock Server for HTTP/TCP Testing
Grab built Loki, a dynamic mock server written in Golang that simulates backend services on local developer machines and CI pipelines. Mobile app testing previously suffered from heavy dependencies on complex, brittle staging environments and interconnected services communicating over HTTP, HTTPS, and TCP. Loki handles both HTTP and TCP traffic on distinct ports while exposing a unified RESTful API to manage test expectations. It provides runtime flexibility through sandboxed JavaScript execution, configurable request sequence ordering, and an in-memory cron scheduler for TCP push messages. Adopting Loki decoupled mobile releases from staging stability, improving delivery cycles and enabling automated UI testing with Espresso and XCUITest.
Saved by the Schema: Using JSON Schema to Document, Test, and Debug APIs
Heroku uses JSON Schema to define Platform API response structures, generate documentation, and validate endpoint behavior. A CI-invoked rake task detects schema changes and uses prmd to generate Markdown API reference material, while committee tests API responses with assert_schema_conform. Those tests exposed an edge case in Enterprise usage data: dyno values with many decimal places were parsed by Oj as BigDecimal and serialized as quoted strings, whereas integer and ordinary float values remained numbers. Heroku traced this behavior through MultiJson and configured Oj with bigdecimal_load: :float, because the endpoint had sufficient precision at 15 decimal places and its schema required a number. The conforming test then passed, confirming that the schema and generated documentation matched returned API values.
How We Harnessed the Wisdom of Crowds to Improve Restaurant Location Accuracy
Grab discovered that abnormally short driver wait times often indicated restaurants registered at incorrect coordinates due to moves or onboarding errors. To fix this, Grab used driver-partner GPS pings, timestamps, and order status updates to infer true food collection locations. The system cleans the data by filtering low-quality GPS pings and isolating the longest temporal streak a driver spends within a predefined radius of the venue. Clusters of inferred pick-up points are then ranked by order volume, the proportion of off-target pick-ups, and median distance errors before routing to mapping operations for verification. This periodic correction workflow achieved a fivefold reduction in order cancellations caused by unfound merchant locations.
Recipe for Building a Widget: How We Helped to “Peak-Shift” Demand by Helping Passengers Understand Travel Trends
Transport demand spikes during regular commuting hours often outpace driver availability, resulting in passenger wait times and fare surges. To mitigate these imbalances, Grab created the Travel Trends Widget for its mobile feed to redistribute ride requests toward off-peak windows. The widget uses machine learning forecasting to present historical supply-demand patterns alongside pricing trends for the upcoming two hours. To handle anticipated high query rates across millions of database entries, engineers periodically load precomputed trend data into an in-memory data structure rather than querying the database per request. The feature rolled out to feeds in Singapore and Jakarta within four weeks of initial development.
Structured Logging: The Best Friend You’ll Want When Things Go Wrong
Grab redesigned its backend logging approach to address mounting vendor costs, query language limitations, and debugging difficulties across a growing microservices ecosystem. Most services previously emitted syslog-style key-value logs almost entirely at the INFO level, which made volume reduction difficult and lacked causal ordering and automated correlation. The engineering team migrated to a self-managed Elastic stack backend and built a structured logging library in Go from the ground up. This framework introduces dynamic log-level adjustment at runtime, automatic trace-based log correlation via Grab-Kit, and the Common Grab Log Schema to enforce consistent JSON formatting without Elasticsearch indexing conflicts.
How We Simplified Our Data Ingestion & Transformation Process
Grab evolved its real-time data ingestion pipeline after an initial architecture built on Spark Streaming and Python encountered operational complexity, node failures, and data loss from S3 eventual consistency. Because the streaming workload primarily handled event partitioning and ORC file generation, the team consolidated these tasks directly into an existing Golang processing service. They implemented sharded concurrent maps for high-throughput partitioning and optimized heap allocations to resolve memory bottlenecks. This refactor removed intermediate Avro conversions and intermediate storage hops. The simplified Go pipeline eliminated data loss and reduced processing lag from up to 13 minutes down to approximately 1 minute.
Deconstructing the Monolith - Shopify
Shopify describes why its decade-old Ruby on Rails monolith became difficult to change as the codebase and development organization grew. High coupling and absent boundaries made innocuous changes trigger unrelated test failures, slowed CI, and forced developers to absorb excessive domain context, while microservices would add deployment, infrastructure, network, and coordination costs. The chosen path was a modular monolith: retain one codebase and deployment unit while separating domains through strictly enforced component boundaries. Shopify’s Componentization effort was informed by a developer survey and uses Wedge to score components and report violations, including cross-component associations and calls that bypass explicitly public interfaces. Boundary enforcement at runtime remains a future goal, but dependency isolation has already enabled replacing a legacy tax engine with a new tax calculation system.
Engineering a Historic Moment: Shopify Gets Ready for Cannabis in Canada
Canada’s 2018 cannabis legalization required Shopify to support a new, tightly regulated retail industry while keeping personal information in Canada. Using Google Cloud Platform’s Montreal region, the team created a Canadian Shopify instance with Terraform-managed projects, regional clusters, segregated workloads, PCI-compliant payment infrastructure, and a regional data warehouse. Because consumer behavior and peak demand were uncertain, engineers modeled traffic scenarios, ran repeated load tests across storefront actions, tested bot attacks, held pre-mortems, and provisioned five times projected launch traffic and sales. Multiple safeguards—including application-level endpoint checks, firewalls, limited secrets, unique SSL certificates, monitoring, and alerting—were layered to prevent data from crossing jurisdictions. Merchants opened across time zones, and the first legal sale occurred on Shopify’s point of sale in Newfoundland at 12:01 NST, followed by launches with zero downtime, excellent performance, and secure Canadian data storage and transmission.
Attracting Local Talent And Building Mobile Apps: A Developer Hiring Initiative
Shopify’s “Build Things, Show Shopify” initiative used app development to reach local mobile developers beyond traditional recruitment channels and strengthen its talent pipeline. From May to August, workshops in Ottawa supported 75+ participants with help from more than 20 Shopify product managers, developers, designers, and computing educators, covering workflows, coding, code reviews, design, and presentation skills. At the September showcase, 13 developers presented 12 apps through rapid three-minute rotations, after which six promising projects advanced to a grand panel of Shopify leaders who provided technical questions and feedback. Participants left with fully developed apps, while a later Ottawa showcase connected them with angel investors and technical recruiters. The initiative also exposed a retention and inclusion problem: participation fell from 24 women among 77 at launch to 16 at the last event, with only two reaching the showcase, prompting consideration of daytime workshops.
A Lean and Scalable Data Pipeline to Capture Large Scale Events and Support Experimentation Platform
Controlled online experimentation across diverse product verticals requires tracking interactions across systems to prevent local optimizations from causing global degradation. Grab built a batch data pipeline to capture, ingest, and process petabytes of event data to support its experimentation platform and analytics stakeholders. The architecture loads ingested event data from Amazon S3, transforms and sorts it, and writes partitioned output back to S3 with metadata registered in Apache Hive. Using Apache Spark on AWS Elastic MapReduce with Apache Airflow for orchestration, the system handles roughly 400,000 incoming events per second. The data is partitioned by event type and ingestion time and stored in Apache ORC format to streamline query workloads and reduce retrieval overhead.
The Unreasonable Effectiveness of Test Retries: An Android Monorepo Case Study
An Android monorepo expansion and larger team increased the number of tests run on master, exposing how flaky tests and infrastructure failures could erode CI confidence. Under an independence assumption, tests passing 99.95% each produced a theoretical full-suite pass rate of only 35% across 20 pipelines, while retrying each failed test once raised it to 99.95%. Shopify therefore added retries for setup failures, distinguishing failures retriable within a job from fatal failures requiring the test environment to reload, and retried failed tests up to three times. The system also flagged tests that passed after retry as warnings, since retries can hide reliability problems and slow builds. In practice, Android app pipeline pass rate rose from 31% to almost 90% after retry mechanisms and Gradle caching, while iOS rose from 67% to 97%.
Querying Big Data in Real-time with Presto & Grab's TalariaDB
Grab developed TalariaDB to support real-time SQL querying over high-velocity event streams while maintaining predictable sub-second latencies and low infrastructure costs. The distributed time-series store retains only the most recent hour of data and integrates directly with Presto via its PrestoThriftService interface. Internally, TalariaDB uses the Go-based Badger key-value store to maintain an in-memory key index of metric names and timestamps while mapping columnar event payloads directly to disk. Ingestion occurs by processing pre-partitioned event batches written to Amazon S3 via SQS notifications. By combining a zero-copy decoder with parallel split evaluation across gossiping cluster nodes, the architecture scales horizontally while serving millions of events per second.
How an Intern Released 3 Terabytes Worth of Storage Before BFCM
While preparing for Black Friday Cyber Monday (BFCM), Shopify’s Products team found that historical reporting records were causing MySQL to scan many InnoDB pages to return a single record. An intern built a Rails background job to delete the records safely, using Shopify’s job-iteration library to process small batches and an internal throttling enumerator to pause when database health or replication lag became a concern. Indexed fetch queries and cursor-based batching avoided repeated scans and timeouts, while the iterator deleted each batch. Run about two weeks before BFCM, the job deleted more than 5 billion records, peaked at approximately six million records per minute, and released three terabytes of storage. Afterward, the database scanned around three times fewer pages per returned record, improving resilience to flash-sale traffic and producing large cost savings.
Orchestrating Chaos Using Grab's Experimentation Platform
Grab operates hundreds of microservices where failures in non-critical components can cause outages in critical user flows if fallback mechanisms are improperly configured. To validate system resilience, Grab built Chaos ExP by layering a chaos engineering SDK and dedicated web UI on top of its existing Experimentation Platform. Integrated directly into the Grab-Kit server middleware, the framework intercepts incoming requests and evaluates whether to inject failures using local variable resolution. Supported failure primitives include latency, errors, panics, rate throttling, and resource leaks to test dependent services. Combining chaos testing with experimentation telemetry enables engineers to correlate injected infrastructure disruptions with business metric impacts.
Top topics
- Architecture834
- Performance817
- LLMs673
- Machine Learning622
- Open Source568
- AI Agents564
- AI483
- Developer Experience448
Weekly digest
The best engineering reads, once a week.