Loading…
Browse by type
Problem & solution reads
Problem & solution posts on Yomu, summarised newest first.
- 934 posts
- 20 companies
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Problem & solution posts
How to Go from a Quick Idea to an Essential Feature in Four Steps
Grab engineered an in-app messaging platform, GrabChat, to help drivers and passengers coordinate pickups across Southeast Asian markets characterized by weak 2G connectivity and high packet loss. The team developed an in-house TCP messaging architecture consisting of a TCP gateway named Gundam and a message dispatcher named Hermes connected to internal backend services over HTTPS. To protect backend server resources from resend loops during poor connection states, the communication protocol adopts a "server only push once" model that delegates retry handling to the client. Data science evaluations using a pre-trained cancellation prediction model confirmed that GrabChat adoption correlated with reduced booking cancellations. Following early usage feedback, the team further iterated on the feature by introducing pre-written message templates to reduce driver distraction on the road.
Troubleshooting Unusual AWS ELB 5XX Error
Grab experienced intermittent HTTP 5XX alerts when its Gothena service sent driver location updates to the Astrolabe service through an AWS Elastic Load Balancer (ELB). CloudWatch metrics revealed that requests were failing to reach healthy backend instances because of an uneven load distribution favoring a single ELB node in one Availability Zone. The team verified that Route 53 was properly using Alias records and ruled out OS-level DNS caching since Linux does not cache DNS queries by default. Connection inspection with netstat across multiple Go services confirmed a heavily skewed distribution of connections toward specific ELB IP addresses. Comparative tests with cURL, tcpdump, Go, Python, and Ruby in an isolated environment demonstrated that Go reused connections across requests while other runtimes opened new connections per request.
Hello RedBeat: A New Celery Beat Scheduler
Heroku Connect relied on Celery Beat to trigger frequent periodic tasks while synchronizing Salesforce and Heroku Postgres data, but the default file-based scheduler developed intermittent pauses and hangs that required manual intervention. Existing SQL-backed scheduling introduced blocked transactions and pauses from periodically saving and reloading the full schedule, while an existing Redis-backed package retained similar reload behavior. RedBeat moves the active schedule and runtime metadata into Redis, allowing updates through Redis keys that Beat reads on its next tick without pausing or reloading. It also uses a lock to prevent concurrent Beat daemons, supports schedule updates from other languages, and has run in production for more than a year without recurring scheduler-related incidents.
Scaling Like a Boss with Presto
Grab experienced severe performance degradation, long queue times, and connection timeouts on its Amazon Redshift analytics cluster as user concurrency and reporting workloads expanded. Although an initial Amazon S3 data lake decoupled storage from compute, business users required standard SQL interfaces rather than Spark data pipelines. The team deployed Presto clusters on AWS EMR, switching their storage format from AVRO to Parquet to support ANSI SQL querying directly against S3. Utilizing a shared Hive metastore on Amazon RDS allowed Grab to adopt a shared-data multi-cluster architecture that isolated distinct workloads across dedicated compute clusters. This setup enabled rapid cluster scaling, streamlined failover, and matched Redshift performance on partitioned time-range queries.
Deep Dive into iOS Automation at Grab - Integration Testing
Grab replaced third-party cloud CI services with an in-house Xcode Server pipeline to run automated iOS integration tests across an expanding engineering team. Instead of performing fresh repository clones on shared cloud instances, Xcode Server pulls code incrementally and mirrors local developer configurations. The team automated testing bots through Ruby scripts and Arcanist hooks tied to Phabricator code reviews, dynamically provisioning bots on diffs and deleting them after merges. This architecture preserves step-by-step UI test execution logs and screenshots for easier failure diagnostics. To maintain stability on on-premises hardware, pre-integration scripts clear application cache states using simctl commands.
Why you should co-locate your Xcode tests
Xcode projects with unit tests commonly separate implementation and test files into two directories, often mirroring the source structure in the tests directory. The post argues that this arrangement creates maintenance overhead, slows navigation, obscures how well-tested a codebase is when browsed or reviewed, and can leave tests with lower coding standards. It proposes co-locating each unit-test file beside its implementation file while retaining the test target association, accepting more files per directory in exchange for clearer relationships and easier movement between code and tests. The same SwiftLint configuration is used for implementation and test code, treating tests as first-class code. Tests without a single-file correspondence, including integration or UI tests, should remain in separate directories and may warrant separate targets; the authors say their codebase has adopted the style.
This Rocket Ain't Stopping - Achieving Zero Downtime for Rails to Golang API Migration
Grab transitioned its public passenger app APIs from a legacy Rails application to a Golang service-oriented architecture to consolidate its codebase and engineering teams. Initial attempts to proxy traffic through a cloned Rails server via gRPC were abandoned after encountering TCP load imbalances during autoscaling events and memory leaks in the gRPC Ruby gem. The team pivoted to direct logic migration, porting Ruby logic directly into Go while decomposing modules into standalone services. Verification relied on log-based load testing and live shadow testing, where write operations were safely validated using mock data access layers that evaluated expected database outcomes. Production rollout progressed endpoint-by-endpoint using requests-per-second traffic throttling and prewarmed AWS Elastic Load Balancers before executing the final DNS switch.
Grab You Some Post-Mortem Reports
Grab describes how its service-oriented architecture makes cross-team production debugging difficult because engineers may lack familiarity with other teams’ code and architecture. Post-mortem reports are presented as the glue for shared understanding, but a sample incident entry lacks service context, bug detail, impact, chronology, and investigation steps. The proposed standard has four requirements—Chronology, Context, Empowerment, and Solutions—covering timelines, service internals, blameless educational diagnosis, and remedies across People, Product, and Process. A template includes initial symptoms, a timestamped timeline, log-based investigation with code evidence, temporary and permanent fixes, and improvement actions, while peer review by another team is required before finalization.
The Curious Case of the Phantom Instance
Datadog dashboards for the grab_attention cluster displayed periodic 1.5X step increases in Elastic Load Balancer (ELB) health check requests and ElastiCache Redis connections, creating the illusion of an untracked instance running outside Auto Scaling Group records. Inspecting instance hostname tags revealed that the existing two instances were simply receiving elevated ping counts from the load balancers. AWS Support clarified that ELB scaling events provision new nodes while keeping old nodes running for roughly 90 minutes to handle cached DNS clients. Investigation also revealed that two separate ELBs were attached to the cluster, altering expected request baselines. Furthermore, the application's health check endpoint initiated a non-pooled Redis connection on every request, directly translating load balancer pings into database connection spikes.
How We Migrated to Active Record 4
Heroku Platform API had remained on Active Record 2.3.18, released in March 2013, creating maintenance and security burdens as support ended and custom query infrastructure replaced newer features. The team moved first to Active Record 3.0.20, preserving 2.x APIs, then directly to 4.2.4, using a feature branch, console-driven fixes, automated and acceptance tests, private deployments, staging, and a partial production rollout. The first step took one engineer about two weeks, while the second took two engineers about two weeks; deployment had no downtime but briefly required a rollback for an acts_as_paranoid bug. Performance improved 18% from 2.3.18 to 3.0.20 but worsened 13% from 3.0.20 to 4.2.4, challenging micro-benchmark expectations. The upgrade ultimately left performance at the same overall level, removed patches, restored supported status, and made future updates and training easier.
Improved production stability with circuit breakers
Heroku’s API consumes more service endpoints than it serves, making its availability dependent on public, internal, and third-party services that can produce tail latency, 500 errors, broken sockets, or bad deploys. To contain these failures, the team wrapped external calls with Redis-backed circuit breakers that count errors over a time window, trip at an error-rate threshold, and support fallback responses or user-friendly exceptions. CB2 provides this implementation, including individually identified breakers, while percentage-based thresholds avoid the scaling problem of fixed error counts. Deployment starts with logging-only breakers so parameters and unexpected internal-API behavior can be reviewed before traffic is affected, followed by monitoring state changes with safeguards against duplicate notifications caused by race conditions. The post also recommends explicit, low network-level timeouts, arguing that slow services can damage stability more than fast failures.
Django and Node together on Heroku
Django-based Heroku Connect needed realtime dashboard support through socket.io and Node.js, but Heroku apps could route traffic through only one dyno type. The team chose to run both stacks in one dyno using a runit buildpack and its sub-Procfile format: Node binds to PORT, while node-http-proxy forwards normal web requests to Gunicorn-hosted Django on 127.0.0.1:$DJANGO_PORT. This arrangement increases contention because processes share memory, CPU, and a filesystem, and it makes dyno logs less directly attributable; the team also flattened Procfiles into Procfile.local for Foreman-based local development. After running the architecture in production for a few months without issue, the team reported minimal proxy overhead and retained single-git-push deployment while establishing a path for websocket communication.
How to Secure the Celery Task Queue with Fernet
Celery jobs sent through a Redis broker can expose their contents because Redis does not currently support SSL and Heroku add-ons communicate over the public web. The proposed solution encrypts jobs before they travel between dynos and Redis by wrapping Kombu’s default serializers with Fernet symmetric encryption from Cryptography. The open-source kombu-fernet-serializers library supplies these encrypted serializers through Setuptools entry points; setup requires adding the package, storing a Fernet key in KOMBU_FERNET_KEY, and configuring Celery to use fernet_json with MIMETYPE in CELERY_ACCEPT_CONTENT. The example creates a Celery app with REDIS_URL as its broker and uses the serializer settings, retaining Redis’s straightforward configuration while addressing sensitive information in transit.
Heroku Connect: Event-driven Data Sync For Force.com
Heroku Connect synchronizes data in both directions between Salesforce (Force.com) and Heroku Postgres, but its original polling design trades simplicity and reliability for synchronization latency, wasted API calls, and unnecessary database queries. The post presents an event-driven alternative using the Force.com Streaming API and PostgreSQL’s LISTEN/NOTIFY system. For Salesforce, Heroku Connect automatically creates a PushTopic per synchronized table, subscribes with Node.js and NForce, and dispatches Redis tasks when notifications arrive. Those events act as change signals rather than complete data transfers: workers query the Force.com SOAP API because streaming has limitations and subscribers may miss events. On Postgres, PL/pgSQL trigger functions call pg_notify after DML, while node-postgres listens on named channels; the system supports added UPDATE and DELETE triggers, though triggers are still required.
Top topics
- Architecture834
- Performance817
- LLMs673
- Machine Learning622
- Open Source568
- AI Agents564
- AI483
- Developer Experience448
Weekly digest
The best engineering reads, once a week.