Loading…
Browse by type
Problem & solution reads
Problem & solution posts on Yomu, summarised newest first.
- 891 posts
- 20 companies
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Problem & solution posts
Evolving the Backend Storage for Platform Metrics
Heroku migrated MetaaS, its internal Metrics as a Service platform metrics backend, from self-hosted Apache Cassandra to AWS’s DynamoDB. MetaaS collects application observations, aggregates them into per-application, per-minute statistics such as median, maximum, and 99th-percentile response time, and stores time-series data for dashboard metrics, alerting, and autoscaling. The migration first duplicated Kafka stream-processing writes into DynamoDB, then used “Science Mode” to compare Cassandra and DynamoDB read results in production without changing returned data. After edge cases were fixed and results matched, retention-period TTL expiry avoided a separate historical data transfer, while gradual query cutovers provided another safeguard. Since 100% of MetaaS queries moved to DynamoDB in May of last year, DynamoDB has remained reliable at scale and been faster and cheaper than the Cassandra clusters, with no noticeable effect on users.
No version left behind: Our epic journey of GitLab upgrades
Grab's self-hosted GitLab instance fell approximately 14 months behind official releases, serving thousands of engineers and monorepos reaching up to 39TB in total footprint. To address security updates and system stability, the team established a structured upgrade routine using Terraform, Packer, and Ansible across GitLab's 5,000-user reference architecture. They implemented staged deployments, upgraded stateful Gitaly nodes via in-place rotation, and resolved primary node clustering bottlenecks under Praefect. By utilizing official GitLab Upgrade Paths and parallelizing component deployments, the team caught up on 24 months of releases in 11 months and reduced release lag from 396 days down to 35 days.
Ensuring data reliability and observability in risk systems
Grab's in-house risk management platform, GrabDefence, relies on ingesting large volumes of upstream data to drive real-time heuristic rules and data science models. To prevent data discrepancies and missing information from degrading fraud detection, the team implemented a real-time data observability pipeline. The solution uses Apache Flink SQL alongside custom table functions like JSONEXPLOAD to flatten nested JSON structures and aggregate metrics over 5-minute tumbling windows. Processed metrics are sent to Datadog, where counters are grouped by source stream and evaluated using anomaly detection algorithms. Alerts are routed directly to Slack, reducing anomaly detection latency from days or weeks down to within the same day or hour.
Grab Experiment Decision Engine - a Unified Toolkit for Experimentation
Grab's Experimentation platform tests thousands of experimental variants weekly, but analyses were largely handled manually using disparate tools unsuited to multi-sided marketplace nuances. To standardize workflows and eliminate reliability issues, Grab developed the GrabX Decision Engine, an internally open-sourced Python toolkit integrated with its Automated Experiment Analysis application. The framework includes a Trusted Advisor component that automates power analysis and sample size estimation while adjusting for clustered standard errors across metric data types. It also bundles a post-experiment statistical toolbox executing default and custom evaluations, ranging from non-parametric tests to regression models that control for confounders and fixed effects. This unified platform institutionalizes experimentation best practices across teams, streamlining collaboration and ensuring consistent statistical inference.
Iris - Turning observations into actionable insights for enhanced decision making
Standard infrastructure monitoring tools track CPU and memory usage at the host or container level, obscuring the precise resource consumption of individual Spark jobs on shared instances. To achieve granular observability, the Iris platform gathers job metrics directly at the Java Virtual Machine level without requiring changes to user application code. Data collection relies on Uber's JVM Profiler for deep JVM, thread, and memory pool diagnostics alongside sparkMeasure for stage and task execution statistics. Collected metrics route asynchronously through an Apache Kafka queue to avoid execution bottlenecks, feeding a Telegraf, InfluxDB, and Grafana stack for real-time monitoring while archiving to a data lake for offline analytics. This multi-tiered telemetry enables engineering teams to identify over- or under-provisioned jobs and optimize ETL resource allocations.
Macramé: Untangling the Knot on the Etsy Android Listing Screen
The Etsy Android listing screen had accumulated nearly 200 experiments in three years, leaving a 4,000-plus-line Fragment, inconsistent patterns, low test coverage, and costly, brittle tests. Macramé replaced that legacy design with immutable state propagated through StateFlow to a reactive UI, transforming screen data into smaller ListingUiModels rendered by RecyclerView and ListAdapter. Sealed Events move through a single Dispatcher to focused Handlers, which update State, start asynchronous work, or emit SideEffects for operations such as navigation and logging. After a five-month rewrite, the new screen’s Time to First Content fell 18%, from 1,585 ms to 1,298 ms, while listings viewed rose 2.4% and business-logic test coverage reached 76%. The architecture still has wiring boilerplate, debugging challenges around Handler interactions, and possible overkill for simpler screens, but its use expanded to Cart and Shop.
JIT WireGuard
Fly.io describes how its gateways provisioned WireGuard peers for flyctl and why the old design became unreliable and slow. NATS delivery losses caused failed provisioning, while CI-generated peers accumulated because gateways never removed them, leaving hundreds of thousands of stale kernel entries that slowed reloads and contributed to kernel panics. The replacement detects handshake-initiation packets with BPF or a WebSockets packet hook, runs enough Noise cryptography to recover the initiator’s public key, then fetches matching peer data over an internal HTTP API and installs it on demand. A rate-limited SQLite cache supports cleanup jobs, and the gateway can use the observed four-tuple to initiate back toward flyctl and accelerate handshakes. After weeks in production, stale peers rounded to none, with faster setup and rebooting that no longer required reloading unused entries.
How We Built The Deals Tab in Swift UI
Etsy’s App Updates team rebuilt the long-standing Updates tab as Deals, a destination for coupons and sales, while targeting an October launch ahead of Cyber Week 2023. The team chose Swift UI and Tuist to create a modular, preview-driven implementation rather than extend legacy Objective-C and server-driven layout code. The screen is assembled from typed API modules, parsed with Decodable, and rendered in a Swift UI List using reusable components such as buttons, headers, footers, and listing cards. To bridge incompatible legacy listing-card code, the module accepts a view builder from the UIKit hosting controller, allowing existing cards to be wrapped while newer cards could be implemented in Swift UI. After three months, Deals shipped ahead of deadline; new modules took one sprint or less, previews decoupled front-end work from API support, and the tab increased session conversion and app visits after its mid-October rollout.
Android App Size at Scale with Project Bonsai
Grab launched Project Bonsai to optimize the size of its Android superapp, which encompassed over four million lines of code across 1,500 modules. Because large app sizes negatively impact user acquisition on low-end devices with constrained bandwidth, the engineering team structured its optimization strategy around measurement, reduction, and containment. The team developed App Sizer, a custom CI-integrated tool that tracks binary compositions and module size contributions in Grafana. Initial reductions targeted Java and Kotlin dex files, resolving bloated transitive R classes by upgrading the Android Gradle Plugin to eliminate overly broad R8 retention rules. These initiatives achieved a 26% reduction in app download size while simultaneously decreasing overall disk footprint.
Increasing velocity by modernizing a Python codebase
An expanding monorepo of machine learning workflows became difficult to change safely as contributors and projects multiplied, undermining the velocity it was intended to support. The team replaced requirements.txt with Poetry, using pyproject.toml for direct dependencies and poetry.lock to record transitive dependencies and hashes, while virtual environments simplified setup. Ruff standardized formatting, linting, and CI enforcement; pytest addressed corner cases and bad assumptions, and mypy added annotations and stricter checks that could expose incompatible changes across call sites. To adopt these controls incrementally, the team used ignore comments and staged remediation, including a script for mypy exceptions, while an ignored revision preserved useful git blame history. The resulting codebase was kept in better shape, enabling faster iteration with greater confidence.
Enabling near real-time data analytics on the data lake
Traditional data lake setups using Parquet on Hive metastores struggle with frequent updates and long pipeline intervals, introducing significant latency for ad hoc queries. Grab solved this by implementing Apache Hudi to support near real-time analytics across bounded relational databases and unbounded Kafka streams. For high-throughput sources, Flink streams Avro log files to Merge On Read tables and generates compaction plans for asynchronous Spark writers. Low-throughput workloads leverage Copy On Write tables, while relational database sources ingest binlogs via Flink Change Data Capture connectors. This architecture reduced analytics data latency to the minute level without overloading production databases and Kafka clusters.
The journey of building a comprehensive attribution platform
Grab needed to modernize its marketing analytics from manual ad hoc queries and high data latency to a platform supporting real-time attribution for pricing models like cost per order. The engineering team initially deployed a pure stream-processing engine using Kappa architecture, Kafka, ScyllaDB, and Redis, which reduced latency from days to minutes and merged ads and promo touchpoints. However, stream-only processing faced high costs, out-of-order event issues, and difficulties running multi-touch models across longer historical windows. Grab transitioned to a Lambda architecture pairing Coban stream processing with Spark-based batch ETL and Amazon S3. This hybrid design separated real-time operational metrics from historical batch reporting, cutting real-time processing costs by approximately 25% while maintaining under 1% data discrepancy.
Event Driven Machines
Event Driven Machines presents Lambdo, a pattern for using Fly Machines as short-lived, event-triggered compute rather than relying on HTTP activation. It polls an SQS queue for JSON events, batches the messages and metadata, then creates fast-booting Fly Machines with the events written to /tmp/events.json; the Machine is destroyed after processing finishes. Application code can come from an existing codebase or Lambdo’s Node and PHP runtime images, with handlers or commands processing each event. SQS message attributes select the required image and can specify VM size and command, while the approach supports arbitrary code and even GPU-enabled Machines, giving each event control over its execution environment.
Rethinking Stream Processing: Data Exploration
Online data materialised in Kafka topics at Grab traditionally remained uninspected until ingested into the Data Lake as offline data, introducing latency and making stream processing applications difficult to develop. To enable earlier data exploration, Grab adopted Apache Zeppelin notebooks integrated with an internal data streaming platform. Zeppelin uses a Flink interpreter to convert interactive SQL queries into Flink jobs submitted to a Flink session cluster. The platform dynamically translates Protobuf schemas into SQL Data Definition Language statements to simplify table discovery. Security and compliance risks are addressed by combining mutual TLS authentication with Strimzi's Open Policy Agent to enforce read-only access policies and audit logging on Kafka topics.
Horizontally scaling the Rails backend of Shop app with Vitess
Shop app’s hockey-stick growth pushed its Ruby on Rails backend and MySQL primary datastore toward capacity, with multi-terabyte storage, week-long schema migrations, and throttled background jobs. After splitting databases and scaling surrounding systems, the team adopted Vitess to shard and re-shard MySQL while retaining SQL compatibility and self-managed primary datastores. They organized user-owned data into a sharded “users” keyspace using user_id, kept other data in an unsharded “global” keyspace, and built application-layer query verifiers to validate routing, distribution, and transaction correctness before migration. The move made schema migrations take hours instead of weeks, removed the reported capacity-driven throttling, and enabled further scaling by adding shards, while introducing concepts such as Sequences, Vindexes, and VSchemas that developers must learn.
Kafka on Kubernetes: Reloaded for fault tolerance
Grab's real-time data streaming platform, Coban, operates Kafka on AWS Elastic Kubernetes Service using Strimzi, allocating an entire EC2 worker node with NVMe instance store volumes to each broker. An initial architecture suffered from client connection errors, broken Network Load Balancer target groups, and zombie Persistent Volume Claims when worker nodes terminated. To achieve automated fault tolerance, the team integrated the AWS Node Termination Handler in Queue Processor mode with Auto Scaling lifecycle hooks, ensuring Kafka receives a SIGTERM to migrate partition leadership gracefully before shutdown. They also introduced the Kubernetes Cluster Autoscaler to dynamically provision replacement nodes during maintenance events and used the AWS Load Balancer Controller with TargetGroupBinding custom resources to dynamically update load balancer targets using IP mode.
Sliding window rate limits in distributed systems
Marketing communications across Grab's user base risked causing notification overload and consent revocations. To enforce personalized daily and weekly frequency caps across more than 270 million users, the team addressed segment membership storage and communication rate limiting. They adopted roaring bitmaps instead of Bloom filters to compactly store user segment data without hash collisions or costly rebuilds upon deletion. For frequency capping, they chose Amazon ElastiCache for Redis over DynamoDB, executing a sliding log rate limiting algorithm directly on the cluster using Lua scripts and sorted sets. Timestamps are stored as sorted set scores, and historical data is cleaned up via eviction ranges to prevent unbounded memory growth.
Rethinking Serverless with FLAME
Rethinking Serverless with FLAME presents a pattern for running selected application functions on short-lived copies of the full app instead of moving code into bespoke FaaS services. FLAME, short for Fleeting Lambda Application for Modular Execution, uses FLAME.call to send a function and closed-over values to a runner pool, while retaining access to the application’s database connections and other code. The Elixir flame library includes LocalBackend and FlyBackend adapters, and Fly.io can boot a same-image Machine in the parent’s region in about three seconds. Runners may stay warm, scale to zero, or grow elastically based on configuration. The post also describes process-based streaming with LiveView and supervision and monitoring mechanisms for transient runners.
An elegant platform
Grab’s Coban real-time data streaming team initially relied on direct Terraform merge requests for managing self-served resources like Kafka topics and CDC pipelines. This pure Infrastructure-as-Code workflow faced challenges with CI pipeline failures from manual errors, security risks from lacking access controls, and platform team review bottlenecks. To resolve this, Coban implemented a three-tier control plane comprising the Coban UI, a Go backend named Heimdall, and a Git storage and provisioner repository named Khone. Heimdall translates user form submissions into validated merge requests containing Terraform and metadata files, polling Khone's CI status and alerting users. This architecture abstracts infrastructure code behind a graphical self-service portal while preserving underlying Git auditing and automation.
Print on Demand
Print on Demand presents a resource-conscious alternative to scaling an application by separating PDF generation from the main server. The post explains how a Fly.io PDF appliance uses Fly-Replay to route a PDF request to a machine running headless Chrome, which requests the original page without its “.pdf” extension and converts the returned HTML into a PDF. Each machine reuses one Chrome instance across requests, while auto_stop_machines allows idle machines to stop and restart on demand. Integration requires cloning the GitHub repository, creating and configuring an app, deploying it, and returning a fly-replay response header for PDF-directed requests. The extracted production code is packaged for deployment with existing applications, supports session, cookies, and basic authentication through passed-back HTTP headers, and can serve applications using different languages and frameworks.
Top topics
- Architecture830
- Performance812
- LLMs648
- Machine Learning617
- Open Source561
- AI Agents540
- AI454
- Developer Experience446
Weekly digest
The best engineering reads, once a week.