Loading…
Browse by type
Problem & solution reads
Problem & solution posts on Yomu, summarised newest first.
- 903 posts
- 20 companies
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Problem & solution posts
Using DNS Traffic Management to Add Resiliency to Shopify’s Services
Shopify consolidated several team-specific DNS traffic-management approaches that relied on manual changes, pull requests, or provider APIs and had uneven failover, redundancy, and active/active capabilities. The replacement was a fully automated, self-served system for more than 40 domains owned by more than 12 teams and handling more than 100M requests per day. It stores domain configuration in Git, deploys it to two DNS providers, and exposes spy commands for fast updates and failovers. Its traffic-management design uses regionalized matching, endpoint-status filtering, weighted sharing, and fallback layers, while globally matching endpoints ensure an existing domain receives an answer. A later provider migration demonstrated the value of centralized ownership: the team updated Terraform and tooling, moved nameservers incrementally, and kept the change transparent to application owners despite concurrent incidents.
How We’re Solving Data Discovery Challenges at Shopify
Shopify describes data discovery as a response to growing numbers of data assets and the difficulty of finding, governing, and using them across the data lifecycle. At Shopify, pre-Artifact discovery relied on people, Slack, GitHub, and job logs, while teams lacked granular ownership, change-impact visibility, and documented reporting context. They built Artifact, a search-and-browse tool backed by a centralized metadata model, ingestion jobs, Elasticsearch indexes, GraphQL APIs, and a graph database for filterable upstream and downstream lineage. Its landing page supports browsing and plain-English search, while details pages expose documentation, schema, descriptions, usage, ownership, and lineage; initial scope prioritized frequently used, high-impact, queryable assets and downstream reports and dashboards. Since launch in early 2020, weekly Data team usage exceeded 30% and monthly retention exceeded 50%, above expectations, with future work planned for additional assets, integrations, stewardship, change management, notifications, and metadata APIs.
Making Time to Save You Time: How We Sped Up Time-Related Syscalls on Dynos
A customer support ticket exposed poor performance in clock_gettime(3) and gettimeofday(2) for an application making many timing calls through transaction tracing. Heroku's Runtime Infrastructure team found that Linux was selecting the Xen clocksource, whose Xen version did not support virtual dynamic shared object (vDSO) acceleration, forcing costly userspace-to-kernel context switches. The team evaluated available clocksources and selected tsc, which uses the Time Stamp Counter and supports vDSO; they assessed concerns about backwards clock drift, including older CPU inconsistencies and Xen VM freezing, against their newer Intel CPUs and EC2 environment. After vdsotest validation, the configuration was deployed across the Common Runtime and Private Spaces dyno fleet, and the initiating customer plus other customers observed performance improvements.
The Journey of Deploying Apache Airflow at Grab
Engineering and data teams across Grab originally operated independent Apache Airflow instances, causing duplicate maintenance overhead and frequent job failures around scaling, logging, and dependency management. To resolve this, a dedicated team developed a centralized orchestration platform that runs isolated, containerized Airflow instances per team on Amazon EKS. The platform categorizes deployments into three size tiers and provisions dedicated Redis brokers, RDS metadata stores, and Vault secret sidecars using Terraform and custom Helm charts. Teams customize container images using shared GitLab CI/CD templates, while worker scaling is handled via Kubernetes Horizontal Pod Autoscalers. Today, the platform runs roughly 20 Airflow instances executing between 1,000 and 60,000 daily jobs per instance.
How We Built Size.link
Size.link is a free Shopify AR tool that lets shoppers view a product’s dimensions in their surroundings through a smartphone camera. It was created during April Hack Days as a quick alternative to commissioning 3D models, with requirements including browser-based operation, speed, accuracy, and iOS and Android support. Because iOS lacks JavaScript WebAR, the implementation uses AR Quick Look with USDZ files and Android’s Scene Viewer with GLB files. Rather than generate files dynamically with slow USDZ tooling or pre-generate billions of dimension combinations, the team modifies binary template data, replacing scale or vertex float values to resize the cube while preserving its outline. The resulting Ruby server generates USDZ files in well under one millisecond, and the same strategy produces GLB files; future work could add textures for standard-sized products.
Media at Scale: Callbacks vs pipelines
Shopify’s Rails monolith needed to add native product videos and 3D models while continuing to support an image infrastructure containing more than 7 billion images. The design question was whether media creation should rely on Active Record callbacks or an explicit pipeline, with transactions protecting interdependent database writes. Callbacks were quick for simple cases, but adding media-specific behavior such as video thumbnails spread conditionals across models and made lifecycle ordering difficult to follow and debug. The pipeline design routes requests through a single Product Create Media Service and media handlers, each organized into before_transaction, during_transaction, and after_transaction steps, while confining logic to one media type. This structure separates concerns, controls creation order, limits model access, and makes implementation details easier to understand and maintain as the feature grows.
How We Built Our In-house Chat Platform for the Web
Grab extended its in-house chat platform to the web to support its internal Customer Support portal. Because the existing TCP gateway only supports unicast connections with one active connection per user, opening multiple browser tabs would repeatedly disconnect previous tabs. Rather than undertaking a complex migration to multicast connections on the server, the team adopted a hybrid client-side strategy using SharedWorker and BroadcastChannel APIs. The implementation uses a SharedWorker to maintain a single WebSocket connection per domain while a BroadcastChannel syncs events across all open tabs. A custom wrapper over the worker manages version transitions during deployments to avoid race conditions across tabs.
Spark Joy by Running Fewer Tests
Shopify’s monolithic Ruby repository has more than 150,000 tests, grows 20–30% annually, and takes 30–40 minutes to run across hundreds of Docker containers, making required full-suite CI costly and vulnerable to intermittent failures. The team built a dynamic-analysis test-selection system that logs method calls during each test, records files in each call graph, and maps changed files to relevant tests. Because Ruby, Rails metaprogramming, and non-Ruby files complicate tracing, the system adds Rails patches and fallback rules, runs extra tests when mappings lag, and executes the full suite asynchronously on every deploy. After two months and 8,360 merged commits, it achieved 99.94% failure recall, selected about 60% of tests, and reduced compute time by roughly 25%. Developers requested full-suite runs on fewer than 2% of pull requests.
Go Modules- A Guide for monorepos (Part 1)
Grab transitioned its large Go monorepo dependency management from Glide to Go modules while retaining an existing vendor directory structure. The team generated root go.mod configurations from glide.yaml and used go mod vendor without directly enabling module-mode builds. Incompatible nested sub-vendor paths were excluded by placing empty go.mod files, relying on the rule that modules cannot contain other modules. Post-migration maintenance revealed challenges with dependency inheritance and implicit go.mod updates during builds, which engineers investigated using go mod graph and digraph to trace dependency paths.
Building and Scaling a Global Chatbot using Heroku + Terraform
Live Agent, Salesforce’s embeddable chatbot, must handle customer interactions and requests at global scale while remaining highly available across regions. Its eight-person team chose Heroku to avoid managing bare metal, cloud virtual machines, databases, and other third-party services, while relying on managed Java support and regional infrastructure. The platform runs ten apps with Heroku Postgres, Redis, and Apache Kafka dependencies, and deploys them inside Heroku Private Spaces to isolate network boundaries and data. Terraform defines and replicates apps, add-ons, custom domains, logging, and profiling configuration across regions, allowing changes to propagate through a pull request. The result described is a simpler operational model in which Heroku handles maintenance and the team can scale the same architecture into new geographic locations.
Tackling UI Test Execution Time Imbalance for Xcode Parallel Testing
Parallel test execution in Xcode can suffer from test time imbalance when tasks finish at significantly different times across parallel simulator workers. Analysis of Xcode scheduling logs shows that the runner groups tests by test class and dispatches all tests from the same class to a single simulator. Attempts to customize the suite by swizzling XCTestSuite fail because made-up suites initialize only after tests are dispatched. To overcome this grouping constraint, unique tokens or test names are appended to the class name component in `-only-testing` command-line arguments. This trick forces Xcode to treat each test as an independent class, successfully distributing individual tests across separate workers.
Refactoring Legacy Code with the Strangler Fig Pattern
Shopify’s Ruby on Rails Shop model had grown beyond 3,000 lines, accumulated numerous responsibilities, and developed weak semantic boundaries, making store settings a target for extraction. The team used Flog’s ABC-based score—assignments, branches, and calls—to identify a particularly disordered area, then applied Martin Fowler’s Strangler Fig Pattern to move the locked_settings behavior into Capital::SettingsToLock and a dedicated record/data source. The seven-step process defined and tested a new interface, redirected callers, introduced and backfilled the new data source, switched reads, and finally stopped legacy writes and removed the old column and code. Keeping the old system available during the transition supported incremental changes, monitoring, reversibility, and the stated no-downtime requirement; the completed example left the new system in place with legacy remnants removed.
Returning 575 Terabytes of Storage Space to Our Users
Android Vitals data revealed that 15.7% of Grab users had less than 1GB of free device storage and uninstalled the app at 1.2 times the normal rate. To understand on-device storage consumption, the team instrumented session launches using the Android StorageManager API to collect binary size, cache folder size, and total footprint metrics. Analysis showed unusually large cache sizes driven by orphaned cache folders from discontinued third-party libraries, including an image library replacement of Picasso by Glide. An automated cleanup routine deployed in app updates purged legacy cache directories upon launch. This mechanism reclaimed 575 terabytes of junk data across more than 13 million devices, averaging 40MB per user.
Developing Zoom Marketplace Apps w/ ngrok
Developers frequently use ngrok to establish fast, introspectable tunnels to localhost when building Zoom Marketplace applications instead of configuring complex reverse proxies like NGINX or Apache. However, using basic ngrok tunnels in production exposes systems to short URL expiration windows, single points of failure, and scalability bottlenecks. To maintain application availability, developers can purchase an ngrok license to secure a service-level agreement and support. Additionally, teams should reserve dedicated subdomains rather than using auto-generated endpoints, run ngrok as a monitored background service on a cloud provider or data center, and place a load balancer in front of the tunnel to handle heavy traffic.
Scaling Mobile Development by Treating Apps as Services
Shopify describes the challenge of scaling mobile development without slowing feature delivery as teams and projects adopt divergent tools, making knowledge sharing, cross-project contribution, and incident response harder. It extends ServicesDB, previously used to track production services, to treat mobile apps as services with owners and explicit expectations encoded in configuration files. Automatic checks cover ownership, contact information, CI and Shipit Mobile deployment configuration, internal-tool versioning, and production error and crash monitoring. Failed checks generate GitHub issues with remediation instructions, while the contract helps the Mobile Tooling team identify compatible projects and helps developers understand why tooling may not work. The implementation currently has basic checks, with planned work including mobile-app service-level objectives, easier project creation, and shorter build times.
How We Prevented App Performance Degradation from Sudden Ride Demand Spikes
Grab experienced severe system strain when sudden localized spikes in ride demand, triggered by events like heavy rain or concert dismissals, coincided with driver shortages. These localized bursts overloaded the platform and degraded the experience for users outside the affected areas. To mitigate this, engineers created the Spampede filter, a circuit-breaker mechanism placed at the start of the booking pipeline. The filter converts pickup locations into Geohash Integer buckets and partitions time using Unix timestamps, tracking unfulfilled requests in Redis with atomic increments and time-to-live expirations. When unallocated requests exceed configured thresholds within a specific bucket, the system immediately short-circuits new incoming bookings to protect overall platform stability.
Plumbing At Scale
Grab's backend services process terabytes of data ingress per hour, generating recurring needs for stream transformations, joins, and time-windowed aggregations across diverse workloads. To support these asynchronous processing patterns across their Go ecosystem, the Coban team developed a managed, NoOps event sourcing and stream processing platform. The architecture packages stateless processing pipelines as Kubernetes deployments on AWS, polling Kafka event logs and using ScyllaDB as a shared metastore for stateful needs like deduplication and windowing. Stream processing pods combine ingestion triggers, a worker pool runtime, and user-provided domain logic plugins with customizable failure handling. This infrastructure scales to handle over 300 billion events weekly while maintaining workload isolation and elastic autoscaling.
Journey to a Faster Everyday Superapp Where Every Millisecond Counts
Grab undertook an initiative to reduce startup time and improve time to interactive (TTI) on its passenger mobile app. Because local benchmarks failed to simulate real device and network conditions, the team instrumented code in production across 8–9 million daily users to capture p50 and p95 metrics. Initial gains came from caching service tiles between sessions and removing a startup animation, saving four seconds. Architectural changes followed, including converting iOS dynamic frameworks to static linking and merging others, while Android initialisation was refactored with Kotlin coroutines. Replacing a heavy third-party analytics library with an internal experimentation platform yielded further startup reductions.
Marionette - Enabling E2E User-scenario Simulation
Conducting end-to-end testing across Grab's transport microservices became difficult due to service availability, environment construction, cross-service authentication, and complex data setups for real-world user accounts. To address these challenges without relying on physical mobile devices or emulators, Grab built Marionette, an internal simulation platform for passenger and driver interactions. The platform provisions required test data, coordinates booking lifecycles, and isolates test executions across distinct user groups using localized cohorts. Engineers can configure driver and passenger behaviors, execute workflows, and run load or integration tests through a dedicated user interface, a Go SDK, and RESTful APIs.
Slow Database Queries? Are Database Types to Blame?
Slow database queries can stem from schema choices as well as inefficient SQL. The example uses PostgreSQL with a json column holding large payloads, then selects a single url value for an app_uuid and orders the results. EXPLAIN ANALYZE with BUFFERS showed 7,106 shared buffer hits, representing about 60MB of JSON data loaded and discarded, while execution reached 1,141.296 milliseconds in the reported run. The solution split the JSON into separate text fields for data and url, bringing query time down from about 1,200 milliseconds to 10 milliseconds. The broader conclusion is that schemas should be revisited as application usage becomes better understood, but changes are not necessary when the design isn't causing a problem.
Top topics
- Architecture830
- Performance813
- LLMs655
- Machine Learning619
- Open Source562
- AI Agents546
- AI459
- Developer Experience447
Weekly digest
The best engineering reads, once a week.