Loading…
Browse by type
Problem & solution reads
Problem & solution posts on Yomu, summarised newest first.
- 934 posts
- 20 companies
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Problem & solution posts
Reliable and Scalable Feature Toggles and A/B Testing SDK at Grab
Grab previously managed experiments using custom service-level code and a toggling library that queried a shared Redis instance, creating latency risks and a single point of failure across backend microservices. To achieve reliable, sub-microsecond feature evaluations, the team designed a Go SDK that resolves rollouts and A/B tests entirely in memory without runtime network I/O. Backend services periodically poll JSON-defined configuration schemas stored in Amazon S3 through a Universal Configuration Manager. The SDK evaluates contextual attributes called facets locally and pushes decision telemetry asynchronously to an S3 and Presto data lake. This architecture allows engineering and product teams to gate deployments and run server-side experiments safely without service disruption.
Cache Invalidation Complexity: Rails 5.2 and Dalli Cache Store
Rails 5.2+ applications can fail to invalidate ActiveRecord-backed cache entries when cache versioning is enabled alongside a third-party store such as dalli_store 2.7.8 or earlier. Rails’ recyclable cache keys keep the cache_key stable and store an ActiveRecord cache_version in the entry, preventing unrelated entries from being evicted, but Dalli only understands the key and therefore repeatedly returns stale data. The post explains how to check the production cache_store and active_record.cache_versioning settings, then compares switching to Rails’ :mem_cache_store, disabling versioning, and upgrading Dalli to 2.7.9+. Its recommendation is to use :mem_cache_store and flush the cache, while noting that Dalli 2.7.9 fixes invalidation by putting the version into the key rather than implementing recyclable keys.
Lost in Translations: Bringing the World to Shopify
Shopify describes the engineering challenge of internationalizing a platform whose twelve years of development left thousands of hardcoded English strings across millions of lines of code. The team first used cops, linters, and tests to stop new violations, then automated extraction of existing content into module-specific dictionaries using keys derived from file paths and strings. A Rails I18n-based pseudolocalization backend replaced characters and doubled vowels, making untranslated content visible while exposing UI problems caused by longer words; UTF-8 migrations and application fixes addressed character-set and internationalization bugs. Finally, an internal Translation Platform connected GitHub webhooks and a translation management system, automatically submitting changed language files and returning approved translations as commits or pull requests. After six months, Shopify was available in a variety of languages, while the team continued addressing localization issues involving formatting, geography, payments, shipping, iconography, and regional requirements.
Mockers - Overcoming Testing Challenges at Grab
Grab operates over 250 microservices communicating over HTTP and gRPC, making shared staging environments costly, ambiguous in ownership, and fragile due to inconsistent data and uncoordinated deployments. To address these testing bottlenecks, Grab created Mockers, a Go SDK and CLI tool backed by a central monorepo of mock servers for local-box and CI testing. Mockers automatically generates HTTP and gRPC mock servers from Swagger specifications and protobuf files, returning configured network responses without internal business logic. By incorporating Grab's in-house chaos SDK middleware, Mockers also enables repeatable resiliency and contract testing locally without relying on code-level mocks. While Grab still mandates integration testing on distributed staging environments with live data, Mockers enables developers to detect complex defects and contract mismatches earlier.
How We Designed the Quotas Microservice to Prevent Resource Abuse
As Grab migrated from a monolith to hundreds of microservices, managing global rate limiting became essential to prevent cascading failures and resource exhaustion. To avoid putting a rate limiting service on the critical path of every API call, Grab built Quotas, an asynchronous rate limiting system. Client services use a lightweight SDK and middleware to read rate limiting decisions from local in-memory caches and stream usage metrics asynchronously via Apache Kafka. The Quotas service aggregates usage data locally, flushes stats to Redis periodically, and publishes updated rate limiting decisions back over Kafka topics. In production, Quotas successfully handles 200k peak transactions per second with decision enforcement delays capped at 200 milliseconds.
Building Grab’s Experimentation Platform
Grab built its internal Experimentation Platform (ExP) to replace a manual, expensive testing process that required bespoke meetings, custom logging pipelines, and service modifications for each experiment. ExP provides a unified infrastructure featuring a centralized management UI, automated real-time data streaming to S3, and SDKs for Android, iOS, and Go. The platform leverages JSON-based experiment definitions delivered through dynamic configuration management, enabling client-side evaluation without costly network calls. It addresses marketplace network effects and inter-experiment interference through mechanisms such as geo-temporal segmentation and domain-layer models. The platform has scaled to run approximately 25 concurrent experiments while computing roughly 2,500 metrics and 50,000 experiment-metric combinations daily.
Scaling iOS CI with Anka
Shopify needed a unified CI/CD system that could give iOS developers the same reliable, disposable build environments already used for Linux and Android projects. An earlier VMware ESXi setup used a shared storage area network, but its maintenance burden and write throughput limit created stability and scalability problems. The replacement uses Anka, a Docker-like macOS virtualization layer built on Apple’s Hypervisor.framework: one Core i7 Mac Mini runs one VM with local storage, while layered images are pushed to registries and pulled across two MacStadium data centers. Buildkite agents clone temporary snapshots, boot suspended VMs in under a second, run steps, upload artifacts, and delete the VMs afterward. Shopify reports about 350 iOS build jobs per hour for roughly 60 iOS developers, with easier Xcode and macOS upgrades, no shared-storage single point of failure, and horizontal scaling, although capacity remains manually adjusted and spiky demand can increase queue times.
Introducing Grab-Kit: Distributed Service Design at Grab
As Grab migrated from a monolith to microservices, maintaining consistency, coordination, and code quality across rapidly expanding teams became a major engineering challenge. To address this, the Developer Experience team built Grab-Kit, a Go framework that automates service scaffolding, code generation, and distributed system design patterns. The framework uses Protocol Buffer definition files as a single source of truth to generate data transfer objects, communication bindings, and standardized middleware for logging and profiling. Grab-Kit also features declarative metrics definitions that synchronize with the DataDog API to build and update service dashboards automatically. Adopting the framework reduced development time for creating new services by up to 70% in teams such as GrabFood while improving overall system stability.
Securing Dependencies for Rails 5.2 Active Storage
Supporting Rails 5.2 Active Storage on Heroku required handling media files for PDF and video previews while addressing FFmpeg’s security and packaging constraints. Ubuntu Trusty’s lack of FFmpeg support, Xenial’s community-maintained package and missing vulnerability backports, and concerns about Libav led the team to build its own binary. Using a fork of sffmpeg and modified CMake configuration, the team created a static FFmpeg build that removed non-free software and unused components such as RTMP and xvid, reducing dependencies and attack surface. Docker-based CircleCI automation compiles the binary, packages it as a Debian archive, uploads it to AWS S3, and lets the buildpack install it without root access. The resulting package is about 30MB instead of roughly 200MB and can be updated in around 20 minutes after upstream security releases.
Solving the N+1 Problem for GraphQL through Batching
GraphQL avoids REST’s over-fetching and under-fetching by letting clients request specific fields, but its field-specific resolvers can create the n+1 problem for nested data. With N authors, a server may make one datastore round trip for authors and N more for addresses, rather than loading all addresses in one additional trip. Shopify’s GraphQL Batch Ruby library addresses this through promises and batch loaders: resolvers request data without resolving immediately, similar loads are grouped, and the loaders fulfill them together. During execution, grouped promises are loaded and replaced with results, while nested fields can enter later batching rounds. The approach reduces datastore queries and computing power for the same requests, became general best practice at Shopify, and was released as open-source.
How Grab Experimented with Chat to Drive Down Booking Cancellations
Post-allocation ride cancellations at Grab degrade the booking experience and create costly inefficiencies for both passengers and driver-partners. Internal user research and platform data confirmed that rides involving GrabChat conversations had significantly lower cancellation rates by reducing perceived wait times. To scale this interaction without extra cost, the team tested system-generated automated messages sent at varying delay intervals, styles, tones, and localized verbiage across different cities. Faster message delivery outperformed longer delays, and tailored prompts reduced booking cancellations by up to two percentage points across tested markets. The experiment demonstrated that high-quality, directed prompts solicited quick responses and improved pick-up efficiency even when overall message volume was lower than control groups.
Accelerating Android Talent Through Community Bootcamps
Shopify created Android Bootcamp to address its need for mobile talent while giving local developers an entry point into Android development. The two-day workshop targeted people with at least one year of development experience, especially newcomers to Canada, and taught participants to build a simple quiz app. The curriculum covered XML-defined views, Android lifecycles, MVC, JSON parsing, libraries, and remote data fetching, with six Shopify mentors supporting one primary teacher. After screening 33 applications, the team accepted 14 people—seven men and seven women—and received positive feedback, although participants found the room too small. Shopify reported 100% overall happiness, one attendee later joined full-time, another was set to join in May, and a second bootcamp was being planned for June.
Future Proofing Our Cloud Storage Usage
Shopify’s growth from 375,000 to over 600,000 merchants coincided with about 450,000 S3 operations per second and rising error rates plus read and write latency. Merchant-uploaded images and theme installations exposed a partitioning problem: S3 groups keys by shortest common asset prefixes, and abrupt throughput spikes could trigger SlowDown failures across a range of shops. Black-box observations suggested affected keyspaces were unlikely to recur, while newer shops were more often involved, though the latter was explicitly described as a shaky inference. Shopify prepended a hash-generated digest to new shops’ asset keys, distributing writes across more partitions instead of concentrating them under shop-specific prefixes. After the first post-change SlowDown event, exceptions nearly disappeared, while median latency for digested assets fell about 60% and the 95th percentile about 25%.
Scaling ipify to 30 Billion Requests and Beyond on Heroku
ipify began as a free API that returns a caller’s public-facing IP address, created to help discover cloud instances’ external addresses without management APIs. Its initial implementation was a sub-50-Loc Node service deployed on one Heroku dyno for about $7/month; rising usage caused slow responses and 503s. Investigation showed roughly 10 rps, while one Node process and string manipulation around X-Forwarded-For limited throughput; clustering doubled capacity, and the implementation later used Go. Heroku supplied load balancing, automatic restarts, and horizontal dyno scaling as traffic expanded through bursty demand and malicious use; the operator chose manual capacity rather than paid autoscaling. The service surpassed 30B requests per month, routinely serving 2k–20k RPS with 1–20ms average response times at $150–$200 monthly; ownership later passed to whoisxmlapi.com, whose work includes an improved UI and more data endpoint.
Grabbing Growth: A Growth Hacking Story
Grab established a dedicated Growth Hacking team within its Technology organization to pursue high-risk, niche initiatives and scale impact across 68 million regional users. Positioning the team within engineering allowed rapid A/B testing and simultaneous multi-market deployments driven by structured growth loops. The team prioritizes and evaluates all initiatives using a Growth Factor metric, calculated as the increase in rides divided by the increase in costs. To improve driver engagement, the team implemented a Spin-to-Win game based on B.F. Skinner's variable ratio reinforcement principles, delivering probabilistic monetary and merchandise rewards upon meeting daily ride thresholds. Ongoing regional experiments continue to evaluate metrics including driver acceptance, cancellation rates, and driver ratings.
Upgrading Kickstarter to Rails 5
Kickstarter upgraded its main Rails application to Rails 5, alongside payments app Rosie, to support site security and performance while moving a nine-year-old legacy codebase forward. The team planned a three-month project, upgraded Ruby 2.2.5 to 2.4.x, classified release-note changes, resolved dependency conflicts, used Rails’ upgrade script, and moved through twelve documented steps. Work included merging Rails 4.2-compatible fixes, repairing nearly 4,000 CI test failures from a roughly 70 percent coverage baseline, removing deprecations, smoke-testing major flows, backporting more than thirty pull requests, and preparing reviewers and deployment contingencies. The upgrade met its timeline without rollbacks or major downtime, although untested bugs increased Zendesk tickets and exposed issues involving ActionController::Parameters, tracking events, and mobile applications. A retrospective emphasized planning, documentation, cross-team communication, and broader native-team participation for future upgrades.
The Data and Science Behind GrabShare Part I: Verifying Potential and Developing the Algorithm
Expanding from point-to-point dispatch services to dynamic carpooling requires matching independent passenger requests traveling in similar directions without causing unacceptable delays. Grab evaluated the feasibility of its GrabShare service by analyzing historical trip data with DBSCAN clustering on coordinates projected into a Universal Transverse Mercator system. This analysis demonstrated that 35% to 46% of rides across typical daytime windows fell into tight geographic clusters with near-identical pickup and drop-off coordinates. The resulting matching framework adapts the baseline dispatch flow by searching for in-transit drivers and enforcing real-time seat reservation constraints. Route assignment decisions subsequently evaluate detour times, trip angles, and expected arrival times to ensure driver utilization improves while total driving distance decreases.
Migrating Existing Datastores
Grab's Identity team faced imminent memory exhaustion on a single Redis node used to cache mobile authentication tokens under rapid user growth. Because read traffic outweighed write traffic by roughly 200 times, the team opted for an AWS ElastiCache cluster with three shards and two read replicas per shard. They executed a zero-downtime, six-phase migration plan while handling a peak load of 20,000 queries per second. The migration transitioned through initial one-time data replication, asynchronous shadow writes, synchronous dual writes, asynchronous read validation, switching primary reads, and final write cleanup. Controlled by feature flags and monitored with metrics at every stage, the migration completed without invalidating tokens or causing service disruptions.
How We Scaled Our Cache and Got a Good Night's Sleep
Growing business load on the Common Data Service (CDS) created potential bottlenecks for its single-threaded Redis cache on ElastiCache, necessitating horizontal scaling for greater capacity and throughput. After ruling out master-slave replication and intermediate Twemproxy setups due to memory constraints and proxy I/O bottlenecks, the team implemented client-side sharding. Using an internal Go package for consistent hashing, CDS instances hash cache keys locally to determine the target shard. The implementation encapsulates hashing inside a thin `ShardedCache` wrapper sharing the original cache interface while supporting Ketama and custom hash functions. Deploying via double-writing cron jobs during off-peak hours reduced database read pressure and improved P99 latency.
DNS Resolution in Go and Cgo
Go applications experiencing load balancing issues across AWS Elastic Load Balancer (ELB) nodes trace uneven traffic distribution to IP address sorting defined in RFC 6724. Comparing Go's native DNS resolver with Cgo and glibc's getaddrinfo shows that both initially sort destination addresses using Rule 9 longest matching prefix rules. Disabling IPv6 on the network interface causes C and Cgo resolvers to return IP addresses in randomized order, while the native Go resolver continues deterministic sorting. Examination of net/addrselect.go reveals that Go's native resolver implements only a subset of the RFC rules and omits dynamic source address selection. Achieving permanent parity requires modifying the Go source code directly.
Top topics
- Architecture834
- Performance817
- LLMs673
- Machine Learning622
- Open Source568
- AI Agents564
- AI483
- Developer Experience448
Weekly digest
The best engineering reads, once a week.