Loading…
Browse by type
Problem & solution reads
Problem & solution posts on Yomu, summarised newest first.
- 934 posts
- 20 companies
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Problem & solution posts
Building and Scaling a Global Chatbot using Heroku + Terraform
Live Agent, Salesforce’s embeddable chatbot, must handle customer interactions and requests at global scale while remaining highly available across regions. Its eight-person team chose Heroku to avoid managing bare metal, cloud virtual machines, databases, and other third-party services, while relying on managed Java support and regional infrastructure. The platform runs ten apps with Heroku Postgres, Redis, and Apache Kafka dependencies, and deploys them inside Heroku Private Spaces to isolate network boundaries and data. Terraform defines and replicates apps, add-ons, custom domains, logging, and profiling configuration across regions, allowing changes to propagate through a pull request. The result described is a simpler operational model in which Heroku handles maintenance and the team can scale the same architecture into new geographic locations.
Tackling UI Test Execution Time Imbalance for Xcode Parallel Testing
Parallel test execution in Xcode can suffer from test time imbalance when tasks finish at significantly different times across parallel simulator workers. Analysis of Xcode scheduling logs shows that the runner groups tests by test class and dispatches all tests from the same class to a single simulator. Attempts to customize the suite by swizzling XCTestSuite fail because made-up suites initialize only after tests are dispatched. To overcome this grouping constraint, unique tokens or test names are appended to the class name component in `-only-testing` command-line arguments. This trick forces Xcode to treat each test as an independent class, successfully distributing individual tests across separate workers.
Refactoring Legacy Code with the Strangler Fig Pattern
Shopify’s Ruby on Rails Shop model had grown beyond 3,000 lines, accumulated numerous responsibilities, and developed weak semantic boundaries, making store settings a target for extraction. The team used Flog’s ABC-based score—assignments, branches, and calls—to identify a particularly disordered area, then applied Martin Fowler’s Strangler Fig Pattern to move the locked_settings behavior into Capital::SettingsToLock and a dedicated record/data source. The seven-step process defined and tested a new interface, redirected callers, introduced and backfilled the new data source, switched reads, and finally stopped legacy writes and removed the old column and code. Keeping the old system available during the transition supported incremental changes, monitoring, reversibility, and the stated no-downtime requirement; the completed example left the new system in place with legacy remnants removed.
Returning 575 Terabytes of Storage Space to Our Users
Android Vitals data revealed that 15.7% of Grab users had less than 1GB of free device storage and uninstalled the app at 1.2 times the normal rate. To understand on-device storage consumption, the team instrumented session launches using the Android StorageManager API to collect binary size, cache folder size, and total footprint metrics. Analysis showed unusually large cache sizes driven by orphaned cache folders from discontinued third-party libraries, including an image library replacement of Picasso by Glide. An automated cleanup routine deployed in app updates purged legacy cache directories upon launch. This mechanism reclaimed 575 terabytes of junk data across more than 13 million devices, averaging 40MB per user.
Developing Zoom Marketplace Apps w/ ngrok
Developers frequently use ngrok to establish fast, introspectable tunnels to localhost when building Zoom Marketplace applications instead of configuring complex reverse proxies like NGINX or Apache. However, using basic ngrok tunnels in production exposes systems to short URL expiration windows, single points of failure, and scalability bottlenecks. To maintain application availability, developers can purchase an ngrok license to secure a service-level agreement and support. Additionally, teams should reserve dedicated subdomains rather than using auto-generated endpoints, run ngrok as a monitored background service on a cloud provider or data center, and place a load balancer in front of the tunnel to handle heavy traffic.
Scaling Mobile Development by Treating Apps as Services
Shopify describes the challenge of scaling mobile development without slowing feature delivery as teams and projects adopt divergent tools, making knowledge sharing, cross-project contribution, and incident response harder. It extends ServicesDB, previously used to track production services, to treat mobile apps as services with owners and explicit expectations encoded in configuration files. Automatic checks cover ownership, contact information, CI and Shipit Mobile deployment configuration, internal-tool versioning, and production error and crash monitoring. Failed checks generate GitHub issues with remediation instructions, while the contract helps the Mobile Tooling team identify compatible projects and helps developers understand why tooling may not work. The implementation currently has basic checks, with planned work including mobile-app service-level objectives, easier project creation, and shorter build times.
How We Prevented App Performance Degradation from Sudden Ride Demand Spikes
Grab experienced severe system strain when sudden localized spikes in ride demand, triggered by events like heavy rain or concert dismissals, coincided with driver shortages. These localized bursts overloaded the platform and degraded the experience for users outside the affected areas. To mitigate this, engineers created the Spampede filter, a circuit-breaker mechanism placed at the start of the booking pipeline. The filter converts pickup locations into Geohash Integer buckets and partitions time using Unix timestamps, tracking unfulfilled requests in Redis with atomic increments and time-to-live expirations. When unallocated requests exceed configured thresholds within a specific bucket, the system immediately short-circuits new incoming bookings to protect overall platform stability.
Plumbing At Scale
Grab's backend services process terabytes of data ingress per hour, generating recurring needs for stream transformations, joins, and time-windowed aggregations across diverse workloads. To support these asynchronous processing patterns across their Go ecosystem, the Coban team developed a managed, NoOps event sourcing and stream processing platform. The architecture packages stateless processing pipelines as Kubernetes deployments on AWS, polling Kafka event logs and using ScyllaDB as a shared metastore for stateful needs like deduplication and windowing. Stream processing pods combine ingestion triggers, a worker pool runtime, and user-provided domain logic plugins with customizable failure handling. This infrastructure scales to handle over 300 billion events weekly while maintaining workload isolation and elastic autoscaling.
Journey to a Faster Everyday Superapp Where Every Millisecond Counts
Grab undertook an initiative to reduce startup time and improve time to interactive (TTI) on its passenger mobile app. Because local benchmarks failed to simulate real device and network conditions, the team instrumented code in production across 8–9 million daily users to capture p50 and p95 metrics. Initial gains came from caching service tiles between sessions and removing a startup animation, saving four seconds. Architectural changes followed, including converting iOS dynamic frameworks to static linking and merging others, while Android initialisation was refactored with Kotlin coroutines. Replacing a heavy third-party analytics library with an internal experimentation platform yielded further startup reductions.
Marionette - Enabling E2E User-scenario Simulation
Conducting end-to-end testing across Grab's transport microservices became difficult due to service availability, environment construction, cross-service authentication, and complex data setups for real-world user accounts. To address these challenges without relying on physical mobile devices or emulators, Grab built Marionette, an internal simulation platform for passenger and driver interactions. The platform provisions required test data, coordinates booking lifecycles, and isolates test executions across distinct user groups using localized cohorts. Engineers can configure driver and passenger behaviors, execute workflows, and run load or integration tests through a dedicated user interface, a Go SDK, and RESTful APIs.
Slow Database Queries? Are Database Types to Blame?
Slow database queries can stem from schema choices as well as inefficient SQL. The example uses PostgreSQL with a json column holding large payloads, then selects a single url value for an app_uuid and orders the results. EXPLAIN ANALYZE with BUFFERS showed 7,106 shared buffer hits, representing about 60MB of JSON data loaded and discarded, while execution reached 1,141.296 milliseconds in the reported run. The solution split the JSON into separate text fields for data and url, bringing query time down from about 1,200 milliseconds to 10 milliseconds. The broader conclusion is that schemas should be revisited as application usage becomes better understood, but changes are not necessary when the design isn't causing a problem.
How We Implemented Domain-Driven Development in Golang
Building GrabPlatform's partner integration self-service portal initially resulted in an unstructured codebase where individual files exceeded 500 lines and lacked proper segregation. Modifying existing functions carried high risks of breaking functionality across imported source collections. To resolve this, the team restructured the Go application using Domain-Driven Design principles in coordination with product domain experts. They mapped business rules into bounded contexts, identified entities and aggregate roots, introduced repository interfaces, and utilized domain events for cross-context communication. The refactoring distributed core functionality evenly, simplified onboarding, and aligned technical terminology with business concepts.
Successfully Merging the Work of 1000+ Developers
Shopify’s single monolith supports changes 40 times a day and roughly 400 commits to master daily, making safe trunk-based development difficult at the scale of 1,000+ developers. Its first Merge Queue limited deployment drift and batch size, but queued pull requests did not receive CI, soft conflicts could break master, and a browser extension enabled accidental direct merges. Merge Queue v2 replaces that workflow with GitHub comments and webhooks, programmatic branch protection, an emergency /shipit --emergency path, and a predictive branch where queued changes receive CI before merging. A reconciliation algorithm synchronizes that branch through GitHub’s GraphQL API, while failure-tolerance thresholds handle flaky tests and remove repeatedly failing pull requests. CI starts when requests enter the queue; batches of eight and three concurrently tested batches balance throughput, risk, and resource use, while the changes improved user experience, deployment safety, and deploy throughput.
Griffin, an Anti-fraud Risk Rule Engine Making Billions of Predictions Daily
Grab's Trust/Identity/Safety team built Griffin, an in-house anti-fraud risk rule engine designed to process billions of daily predictions across multiple business verticals. Initially, Grab managed fraud rules directly within backend service code, but escalating rule complexity, tight rule interdependencies, and translation gaps between data scientists and developers caused deployment delays and misfiring errors. To overcome the limitations and steep learning curves of third-party engines like Drools, the team separated the workflow into data orchestration and rule-based prediction. Griffin enables analysts and data scientists to author Python-based rules directly via a web portal and reload updated logic into memory without manual developer intervention. By eliminating I/O during rule evaluation and leveraging Gunicorn multi-processing, the engine handles over 100,000 queries per second at peak on six EC2 instances with single-prediction latencies under six milliseconds.
Automated Continuous Deployment at Heroku
Heroku’s Runtime team replaced a manual, coordination-heavy deployment process after team growth and Common Runtime sharding expanded work from one infrequently changed application in two environments to multiple services across 12 environments. Existing Heroku Pipelines deploy merged, CI-passing changes to staging, while the Go-based cedar-service-deployer reconciles differences between pipeline stages and promotes them when a series of safety checks pass. Those checks include a 30-minute age gate, deployment and incident controls, master-branch verification, Librato alerts, and HTTP 200 checks for configured URLs, including platform integration tests. The stateless system can handle concurrent rollouts, and the team added practices such as isolated early stages, feature flags, and separated database-migration changes. After about two years, it managed 23 services, allowing engineers to merge and rely on healthy systems for global release.
Using Grab’s Trust Counter Service to Detect Fraud Successfully
Grab's Trust Platform team built the Counter service to detect fraud across business verticals like transportation, food, and payments. The platform replaces manual, multi-week engineering cycles with a self-service UI where data analysts can define and experiment with counters independently. Operating on an asynchronous ingestion and synchronous transaction model, the architecture evaluates incoming stream data, enriches it via internal services, and persists aggregated signals to ScyllaDB through Grab-Stats. A multi-bucket strategy partitions queries into fifteen-minute, hourly, and daily granularities to maintain low-latency query aggregations across wide time ranges under strict SLAs.
How Shopify Manages Petabyte Scale MySQL Backup and Restore
Shopify manages a petabyte-scale fleet of MySQL servers across replica-sets, or shards, in three Google Cloud Platform regions, where file-based Xtrabackup archives made shard restores take more than six hours. To reduce that Recovery Time Objective, the team built Kubernetes CronJobs around the Compute Engine Persistent Disk snapshot API, scheduling snapshots every 15 minutes while accounting for regions, zones, instance roles, and MySQL consistency variables. Persistent Disk snapshots take about 20 minutes initially and typically under 10 minutes incrementally; restoring the latest snapshot to a new volume and starting MySQL, including InnoDB instance recovery and replication-lag recovery, typically brings RTO below 30 minutes. Retention tooling keeps the latest two snapshots plus policy-defined dailies and weeklies, while verification checks startup, GTID auto-positioning, and InnoDB page corruption; offsite tooling compresses, encrypts, and transfers exported data to another provider.
Building Resilient GraphQL APIs Using Idempotency
Shopify’s Payment Service uses API-level idempotency to make GraphQL payment mutations resilient to timeouts, connection failures, and ambiguous outcomes that could otherwise leave merchants unpaid or double-charge buyers. The design places a UUID-based idempotency key directly in each mutation input, identifies requests by client plus key, and locks concurrent duplicates so retries receive a 409 response while the original call is processing. An IncomingRequest record stores request state and completed recovery points, allowing completed responses to be replayed and incomplete attempts to recover before continuing. Handlers divide work into no-side-effect, transactional local-side-effect, and remote-side-effect steps, each optionally providing run and recover functions. The approach adds database-write overhead and requires careful compatibility testing when recovery points change, but the team reports that its structured handlers are clear and maintainable.
Designing for Accessibility: Contrast Ratio
Heroku’s accessibility initiative targeted WCAG 2.0 Level AA luminance contrast for its user interfaces, addressing reduced contrast sensitivity and color-vision deficiencies. The team audited its palette with Contrast Grid, proposed darker and revised colors, then chose a narrower proposal that preserved existing names and colors while meeting accessibility concerns within an affordable implementation effort. Because the palette affected GUIs built with Purple 3 and Malibu, pull requests generated Review Apps so teams could audit production-ready CSS and SVG assets across Dashboard components before release. Audits using axe-core 3.2.2 covered common and unusual elements, reducing color-contrast violations by an order of magnitude before rollout. The updated palette now spans the Heroku Dashboard, with planned accessibility testing through Ember tooling and further audits across other interfaces.
Data First, SLA Always
Grab's Data Engineering team transitioned from periodic batch ETL ingestion to a real-time change data capture architecture called Trailblazer after dataset sizes exceeded the petabyte mark. The previous system caused severe JDBC timeouts and heavy CPU loads when executing chunked or full-scan queries on unindexed upstream MySQL tables. To solve this, MySQL binary logs are captured via Debezium on Kafka Connect, buffered in Kafka, and ingested into a data lake using Spark Structured Streaming. Checkpoints are decoupled from local storage and persisted in a Redis cluster to simplify ingestion offset overrides and handle ephemeral compute clusters. The system incorporates extensive health monitoring across Airflow, Datadog, and custom services to maintain stream liveliness and avoid Kafka retention breaches.
Top topics
- Architecture834
- Performance817
- LLMs673
- Machine Learning622
- Open Source568
- AI Agents564
- AI483
- Developer Experience448
Weekly digest
The best engineering reads, once a week.