---
title: "Architecture"
description: "834 posts about Architecture, summarised, each linking to the original."
---

# Architecture
> 834 posts about Architecture, summarised, each linking to the original.

## Articles

### [Building an Event-Driven Architecture with Managed Data Services](https://yomu.fyi/post/building-an-event-driven-architecture-with-managed-data-services.md)
- Company: [Heroku](https://yomu.fyi/company/heroku.md)
- Author: Jonathan Brown
- Published: Aug 15, 2024
- Type: Explainer

Event-driven architecture (EDA) addresses the need to process and respond to continuous streams of real-time activity from applications, e-commerce systems, wearables, and other IoT sensors. Using Apache Kafka, independent producers publish events to topics while consumers subscribe and trigger downstream processes, keeping services decoupled and independently scalable. Kafka also distributes large data streams in real time and replicates data across synchronized broker nodes, so a node failure does not lose data or interrupt operation. Apache Kafka on Heroku is presented as a fully managed service, with multi-tenant basic and single-tenant private plans, automatic broker scaling, and Standard-tier-and-above configurations starting with three brokers and extending to eight. Recommended integration practices include mapping producer-consumer flows, using transactions, topic and schema management, and delivery guarantees, plus monitoring performance and logging Kafka operations.


### [LLM-powered data classification for data entities at scale](https://yomu.fyi/post/llm-powered-data-classification-for-data-entities-at-scale.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Hualin Liu
- Published: Jul 15, 2024
- Type: Problem & solution

Grab needed to classify sensitive data at the table and column level across petabytes of database tables and streaming schemas. Manual schema-tiering campaigns had resulted in half of all schemas receiving overly strict Tier 1 access controls, while an initial automated service using regex patterns and third-party machine learning produced high false-positive rates and lacked customizability. To address this, the Caspian data engineering and governance teams enhanced their internal orchestration service, Gemini, by integrating GPT-3.5 via Azure OpenAI. Gemini aggregates classification requests into mini-batches, handles API rate limits, and uses prompt engineering—including few-shot examples, curated tag libraries, and explicit JSON DTO schemas—to reliably tag columns for data owner verification.


### [Office Hours with Engineering Managing Director Mae Santos](https://yomu.fyi/post/office-hours-with-engineering-managing-director-mae-santos.md)
- Company: [Two Sigma](https://yomu.fyi/company/two-sigma.md)
- Author: Emily Majewski
- Published: Jun 26, 2024
- Type: Explainer

Mae Santos, Head of Critical Applications and Data Reliability Engineering at Two Sigma, leads a global team across New York, Houston, London, and Tokyo to support critical applications and data pipelines. While software teams frequently prioritize functional features over operational stability, designing systems with reliability in mind from the beginning allows for progressive enhancements without complete rewrites. System architecture should be guided by concrete goals across four key dimensions: availability, observability, scalability, and supportability. In addition, mapping out interconnections among systems, infrastructure, data, and human processes helps teams manage dependencies effectively. Applying automation across support and delivery workflows further reinforces reliability by minimizing human error and accelerating execution.


### [No version left behind: Our epic journey of GitLab upgrades](https://yomu.fyi/post/no-version-left-behind-our-epic-journey-of-gitlab-upgrades.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Saurabh Vajpayee
- Published: May 3, 2024
- Type: Problem & solution

Grab's self-hosted GitLab instance fell approximately 14 months behind official releases, serving thousands of engineers and monorepos reaching up to 39TB in total footprint. To address security updates and system stability, the team established a structured upgrade routine using Terraform, Packer, and Ansible across GitLab's 5,000-user reference architecture. They implemented staged deployments, upgraded stateful Gitaly nodes via in-place rotation, and resolved primary node clustering bottlenecks under Praefect. By utilizing official GitLab Upgrade Paths and parallelizing component deployments, the team caught up on 24 months of releases in 11 months and reduced release lag from 396 days down to 35 days.


### [Macramé: Untangling the Knot on the Etsy Android Listing Screen](https://yomu.fyi/post/macrame-untangling-the-knot-on-the-etsy-android-listing-screen.md)
- Company: [Etsy](https://yomu.fyi/company/etsy.md)
- Author: John Kalmi
- Published: Mar 18, 2024
- Type: Problem & solution

The Etsy Android listing screen had accumulated nearly 200 experiments in three years, leaving a 4,000-plus-line Fragment, inconsistent patterns, low test coverage, and costly, brittle tests. Macramé replaced that legacy design with immutable state propagated through StateFlow to a reactive UI, transforming screen data into smaller ListingUiModels rendered by RecyclerView and ListAdapter. Sealed Events move through a single Dispatcher to focused Handlers, which update State, start asynchronous work, or emit SideEffects for operations such as navigation and logging. After a five-month rewrite, the new screen’s Time to First Content fell 18%, from 1,585 ms to 1,298 ms, while listings viewed rose 2.4% and business-logic test coverage reached 76%. The architecture still has wiring boilerplate, debugging challenges around Handler interactions, and possible overkill for simpler screens, but its use expanded to Cart and Shop.


### [How We Built The Deals Tab in Swift UI](https://yomu.fyi/post/how-we-built-the-deals-tab-in-swift-ui.md)
- Company: [Etsy](https://yomu.fyi/company/etsy.md)
- Author: Pierre Rodgers
- Published: Mar 5, 2024
- Type: Problem & solution

Etsy’s App Updates team rebuilt the long-standing Updates tab as Deals, a destination for coupons and sales, while targeting an October launch ahead of Cyber Week 2023. The team chose Swift UI and Tuist to create a modular, preview-driven implementation rather than extend legacy Objective-C and server-driven layout code. The screen is assembled from typed API modules, parsed with Decodable, and rendered in a Swift UI List using reusable components such as buttons, headers, footers, and listing cards. To bridge incompatible legacy listing-card code, the module accepts a view builder from the UIKit hosting controller, allowing existing cards to be wrapped while newer cards could be implemented in Swift UI. After three months, Deals shipped ahead of deadline; new modules took one sprint or less, previews decoupled front-end work from API support, and the tab increased session conversion and app visits after its mid-October rollout.


### [Enabling near real-time data analytics on the data lake](https://yomu.fyi/post/enabling-near-real-time-data-analytics-on-the-data-lake.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Shi Kai Ng
- Published: Feb 23, 2024
- Type: Problem & solution

Traditional data lake setups using Parquet on Hive metastores struggle with frequent updates and long pipeline intervals, introducing significant latency for ad hoc queries. Grab solved this by implementing Apache Hudi to support near real-time analytics across bounded relational databases and unbounded Kafka streams. For high-throughput sources, Flink streams Avro log files to Merge On Read tables and generates compaction plans for asynchronous Spark writers. Low-throughput workloads leverage Copy On Write tables, while relational database sources ingest binlogs via Flink Change Data Capture connectors. This architecture reduced analytics data latency to the minute level without overloading production databases and Kafka clusters.


### [The journey of building a comprehensive attribution platform](https://yomu.fyi/post/the-journey-of-building-a-comprehensive-attribution-platform.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Kang Huang
- Published: Feb 20, 2024
- Type: Problem & solution

Grab needed to modernize its marketing analytics from manual ad hoc queries and high data latency to a platform supporting real-time attribution for pricing models like cost per order. The engineering team initially deployed a pure stream-processing engine using Kappa architecture, Kafka, ScyllaDB, and Redis, which reduced latency from days to minutes and merged ads and promo touchpoints. However, stream-only processing faced high costs, out-of-order event issues, and difficulties running multi-touch models across longer historical windows. Grab transitioned to a Lambda architecture pairing Coban stream processing with Spark-based batch ETL and Amazon S3. This hybrid design separated real-time operational metrics from historical batch reporting, cutting real-time processing costs by approximately 25% while maintaining under 1% data discrepancy.


### [A Packwerk Retrospective](https://yomu.fyi/post/a-packwerk-retrospective.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Feb 7, 2024
- Type: Explainer

Shopify’s retrospective examines Packwerk, a Ruby gem released in September 2020 to enforce package boundaries and help modularize Rails applications. It describes the tool as static analysis that checks constant references against a declared, non-circular dependency graph, with todo files allowing existing violations to be recorded while new ones are blocked. The authors recount how privacy checks shifted attention toward API design, introduced awkward file organization, and were removed in version 3.0. They also explain that Packwerk cannot determine whether package boundaries or violation fixes reflect runtime coupling, so semantic grouping can mislead developers and add harmful indirection. Shopify ultimately treats running code as the strongest progress signal and retains Packwerk mainly to prevent new dependencies at the application’s base layer.


### [Delegating tasks to Fly Machines](https://yomu.fyi/post/delegating-tasks-to-fly-machines.md)
- Company: [Fly.io](https://yomu.fyi/company/fly-io.md)
- Published: Feb 1, 2024
- Type: Tutorial

Resource-intensive web tasks such as resizing high-resolution images can require substantially more memory or CPU than ordinary request handling, making an always-powerful web server inefficient. The article presents a framework- and language-agnostic delegation pattern using Fly Machines, the private network, and two process groups that run the same application with different Machine specifications. A Next.js route calls a wrapper that executes the task when FLY\_PROCESS\_GROUP is worker; otherwise, it sends a POST request to the identical endpoint through the worker’s internal hostname, returning the worker’s response. The example uses a lighter web group and a performance-4x worker group, with local development using separate ports and production routing over .internal addresses; the author calls the implementation naïve and suggests queues and worker timeouts to improve resilience and control worker availability.


### [Kafka on Kubernetes: Reloaded for fault tolerance](https://yomu.fyi/post/kafka-on-kubernetes-reloaded-for-fault-tolerance.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Fabrice Harbulot
- Published: Dec 26, 2023
- Type: Problem & solution

Grab's real-time data streaming platform, Coban, operates Kafka on AWS Elastic Kubernetes Service using Strimzi, allocating an entire EC2 worker node with NVMe instance store volumes to each broker. An initial architecture suffered from client connection errors, broken Network Load Balancer target groups, and zombie Persistent Volume Claims when worker nodes terminated. To achieve automated fault tolerance, the team integrated the AWS Node Termination Handler in Queue Processor mode with Auto Scaling lifecycle hooks, ensuring Kafka receives a SIGTERM to migrate partition leadership gracefully before shutdown. They also introduced the Kubernetes Cluster Autoscaler to dynamically provision replacement nodes during maintenance events and used the AWS Load Balancer Controller with TargetGroupBinding custom resources to dynamically update load balancer targets using IP mode.


### [Introducing Fly Kubernetes](https://yomu.fyi/post/introducing-fly-kubernetes.md)
- Company: [Fly.io](https://yomu.fyi/company/fly-io.md)
- Published: Dec 18, 2023
- Type: Announcement

Fly.io introduces Fly Kubernetes (FKS), an implementation of Kubernetes that runs on its globally distributed container platform for users who want Kubernetes tooling and declarative orchestration. Fly.io says its earlier Nomad-based approach struggled with global scheduling at more than 200,000 applications and with placing workloads in users’ requested regions, so it moved state to individual workers and exposed the Fly Machines API, whose placement requests fail quickly rather than retry elsewhere. FKS combines K3s with Virtual Kubelet, mapping pod lifecycle operations to Fly Machines API calls while running K3s and Virtual Kubelet on one Fly Machine. A generated kubeconfig enables kubectl access, and Pods run on separate Fly Machines as the platform scales the cluster. The design provides declarative replica, autoscaling, and volume definitions plus automatic recovery for failed Machines, while retaining Fly Machines’ trade-off of failing when requested placement is unavailable.


### [Sliding window rate limits in distributed systems](https://yomu.fyi/post/sliding-window-rate-limits-in-distributed-systems.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Naveen Kumar Jakuva Premkumar
- Published: Dec 14, 2023
- Type: Problem & solution

Marketing communications across Grab's user base risked causing notification overload and consent revocations. To enforce personalized daily and weekly frequency caps across more than 270 million users, the team addressed segment membership storage and communication rate limiting. They adopted roaring bitmaps instead of Bloom filters to compactly store user segment data without hash collisions or costly rebuilds upon deletion. For frequency capping, they chose Amazon ElastiCache for Redis over DynamoDB, executing a sliding log rate limiting algorithm directly on the cluster using Lua scripts and sorted sets. Timestamps are stored as sorted set scores, and historical data is cleaned up via eviction ranges to prevent unbounded memory growth.


### [An elegant platform](https://yomu.fyi/post/an-elegant-platform.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Fabrice Harbulot
- Published: Nov 30, 2023
- Type: Problem & solution

Grab’s Coban real-time data streaming team initially relied on direct Terraform merge requests for managing self-served resources like Kafka topics and CDC pipelines. This pure Infrastructure-as-Code workflow faced challenges with CI pipeline failures from manual errors, security risks from lacking access controls, and platform team review bottlenecks. To resolve this, Coban implemented a three-tier control plane comprising the Coban UI, a Go backend named Heimdall, and a Git storage and provisioner repository named Khone. Heimdall translates user form submissions into validated merge requests containing Terraform and metadata files, polling Khone's CI status and alerting users. This architecture abstracts infrastructure code behind a graphical self-service portal while preserving underlying Git auditing and automation.


### [Print on Demand](https://yomu.fyi/post/print-on-demand.md)
- Company: [Fly.io](https://yomu.fyi/company/fly-io.md)
- Published: Nov 29, 2023
- Type: Problem & solution

Print on Demand presents a resource-conscious alternative to scaling an application by separating PDF generation from the main server. The post explains how a Fly.io PDF appliance uses Fly-Replay to route a PDF request to a machine running headless Chrome, which requests the original page without its “.pdf” extension and converts the returned HTML into a PDF. Each machine reuses one Chrome instance across requests, while auto\_stop\_machines allows idle machines to stop and restart on demand. Integration requires cloning the GitHub repository, creating and configuring an app, deploying it, and returning a fly-replay response header for PDF-directed requests. The extracted production code is packaged for deployment with existing applications, supports session, cookies, and basic authentication through passed-back HTTP headers, and can serve applications using different languages and frameworks.


### [Road localisation in GrabMaps](https://yomu.fyi/post/road-localisation-in-grabmaps.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Roxana Crisan
- Published: Nov 17, 2023
- Type: Problem & solution

Grab needed to localise nearly 30 million road segments across more than 10,000 area boundaries in Southeast Asia to attach hyperlocal attributes like language, driving side, and vehicle access rules. Testing direct geometric inclusion between complex road polylines and boundary polygons proved computationally prohibitive for daily map generation pipelines. To optimize this process, Grab introduced geohashes as rectangular proxies, precomputing geohash coverage across borders and road segments before joining the datasets in parallel. To resolve misclassification errors near borders without sacrificing performance, the team evaluated geohash coverage percentages and refined boundary-touching geohashes with polygon clipping. A laptop benchmark on a subset of data reduced runtime from 38 minutes with geometric intersection to 78 seconds using the geohash approximation.


### [Graph modelling guidelines](https://yomu.fyi/post/graph-modelling-guidelines.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Wenxiang Lu
- Published: Nov 8, 2023
- Type: Tutorial

Graph modelling uses graph theory to represent real-world entities, relationships, and properties through nodes and edges, unlocking insights across interconnected datasets. The process begins with defining domain concepts, mapping entities and relationships, and assigning relevant descriptive properties to both nodes and edges. Practitioners choose between graph structures such as property graphs or Resource Description Framework models before optionally developing a structural schema. Data is then imported or generated to populate storage systems such as Neo4j, Amazon Neptune, Azure Cosmos DB, or in-memory data structures. Finally, the graph is visualised and analysed using algorithmic methods like shortest path calculation, centrality measurement, and community detection.


### [Abstraction Engineering](https://yomu.fyi/post/abstraction-engineering.md)
- Company: [Ramp](https://yomu.fyi/company/ramp.md)
- Author: Rodda John, Jared Wasserman
- Published: Oct 31, 2023
- Type: Explainer

Ramp’s early product encoded opinionated limits, but by fall 2022 customers needed configuration based on departments, accounting fields, subsidiaries, budgets, and HRIS fields, making incremental feature work increasingly complex. The team reframed these requests as a generic workflows platform in which Python functions served as actions, boolean logic as conditions, and workflows as graphs persisted in Postgres. After rejecting a dependency-based design, they built a graph-traversal engine, a Postgres-backed queue that retries asynchronous tasks, and an SDK for expressing constructs such as elif and else. The infrastructure now supports customer-facing policies and internal tasks; more than 45 million workflows have run, at over a million per day, while the core engine has not changed since launch. The project took weeks upfront but saved the engineering organization months by turning complex features into applications of broader abstractions.


### [Router 2.0 Common Runtime Router: The Road to Beta](https://yomu.fyi/post/router-2-0-common-runtime-router-the-road-to-beta.md)
- Company: [Heroku](https://yomu.fyi/company/heroku.md)
- Author: Elizabeth Cox
- Published: Oct 30, 2023
- Type: Explainer

Heroku’s Router 2.0 is a beta replacement for the Common Runtime’s Erlang-based “Hermes” router, which is planned to be replaced in 2024. The project addresses the difficulty of safely delivering upgrades from a legacy custom HTTP server and aims to share routing code with the Go-based Private Spaces router; common functionality has been extracted into an internal HTTP library. Architecturally, Router 2.0 is an L7 router running in memory behind the existing L4 proxy, alongside Hermes, allowing app-level traffic to move from the old path without a big-bang cutover. The team added continuous load testing that exercises scaling policies and blocks deployments missing internal service-level objectives, and introduced Network Error Logging to measure browser-observed failures. Feature parity is still incomplete, with HTTP/2 planned next; apps can opt into the beta using a Heroku Labs command.


### [Scaling marketing for merchants with targeted and intelligent promos](https://yomu.fyi/post/scaling-marketing-for-merchants-with-targeted-and-intelligent-promos.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Sharon Teng
- Published: Oct 11, 2023
- Type: Problem & solution

Grab previously relied on globally assigned, heuristic promotional campaigns where all users could redeem offers until limits were reached. This lack of targeting and customisation failed to optimize promo spending or meet specific merchant business objectives. To solve this, the engineering team built Bullseye, an automated AI-driven promotional assignment system that customises and assigns offers to consumers. The architecture integrates an operations user interface, backend services, Amazon S3 storage, and Spark jobs executing a suite of predictive modules including eater segmentation, campaign impact simulation, and customer response modeling. Since deploying Bullseye in 2021, the system has increased food campaign sales while decreasing promo spend across food and Mart campaigns.


[Newer posts](https://yomu.fyi/topic/architecture/page/29.md) · [Older posts](https://yomu.fyi/topic/architecture/page/31.md)