---
title: "Architecture"
description: "834 posts about Architecture, summarised, each linking to the original."
---

# Architecture
> 834 posts about Architecture, summarised, each linking to the original.

## Articles

### [Stepping up marketing for advertisers: Scalable lookalike audience](https://yomu.fyi/post/stepping-up-marketing-for-advertisers-scalable-lookalike-audience.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: William Wu
- Published: Sep 22, 2023
- Type: Problem & solution

Grab's legacy lookalike audience generation platform suffered from long creation SLAs of two working days, high costs, and low weekly update frequencies. To resolve these bottlenecks, the engineering team designed an embedding-based platform powered by an in-memory retrieval service and automated update pipelines. The system creates audience representations by averaging constituent passenger embeddings, determining user membership through real-time cosine score thresholds. To eliminate feature store latency while fitting all embeddings into memory, a hash-based compression method cuts passenger embedding storage needs by roughly 90%. Consequently, audience availability dropped to within 15 minutes of campaign creation, audience generation costs fell by 98%, and ad impressions and clicks doubled.


### [Building a ShopifyQL Code Editor](https://yomu.fyi/post/building-a-shopifyql-code-editor.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Sep 11, 2023
- Type: Tutorial

ShopifyQL Notebooks, released in October 2022, is a first-party app that lets merchants analyze shop data. To give its guided code-editing experience, Shopify used CodeMirror, which did not natively support ShopifyQL. Because ShopifyQL’s grammar is defined in ANTLR and generates Go and TypeScript code, while the language server follows Microsoft’s Language Server Protocol (LSP), the existing client and server language infrastructure was retained. A custom adapter passes queries to the language server, converts its incremental five-integer token descriptors into CodeMirror-compatible offsets, and reshapes them into a buffer of Lezer nodes for parse-tree construction. The resulting integration supports syntax highlighting, code completion, linting, and hover tooltips through CodeMirror plugins, while using its internal parse tree to inform editing behavior.


### [Building hyperlocal GrabMaps](https://yomu.fyi/post/building-hyperlocal-grabmaps.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Adriana Lazar
- Published: Aug 30, 2023
- Type: Problem & solution

Southeast Asia presents unique mapping challenges, including fast-changing road restrictions, new roads appearing rapidly, and severe traffic congestion. To address these conditions, GrabMaps integrates hyperlocal data derived from local team knowledge, rides and deliveries GPS pings, and KartaView imagery with OpenStreetMap data. The pipeline downloads OpenStreetMap .pbf files for Asia and converts them into Parquet format using a Parquetizer. GrabMaps processes spatial data using Apache Spark and bi-directional graphs divided by geohash to detect complex features such as intersection links based on road parallelism, naming, direction, and shape geometry. Tagging these intersection links enables downstream navigation services like GrabNav to calculate shorter routing alternatives and provide more accurate pricing.


### [Streamlining Grab's Segmentation Platform with faster creation and lower latency](https://yomu.fyi/post/streamlining-grab-s-segmentation-platform-with-faster-creation-and-low.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Jake Ng
- Published: Aug 15, 2023
- Type: Problem & solution

Grab's Segmentation Platform previously stored user-to-segment mappings across individual rows in ScyllaDB, causing write bottlenecks during segment creation and read latencies too high for downstream consumers. To resolve these performance limitations, the team transitioned to storing segments as Roaring Bitmaps saved as single blobs in object storage. This compression strategy splits 32-bit integer user IDs into chunks across array, bitmap, and run containers based on data density, reducing a one-million-member segment to under one megabyte. An accompanying client SDK manages segment retrieval, decoding, update notifications, and least-recently-used in-memory caching. Consequently, consumers such as Grab's communications platform achieved peak throughput of 15,000 queries per second with sub-millisecond p99 read latencies.


### [Unsupervised graph anomaly detection - Catching new fraudulent behaviours](https://yomu.fyi/post/unsupervised-graph-anomaly-detection-catching-new-fraudulent-behaviour.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Rizal Fathony
- Published: Aug 2, 2023
- Type: Problem & solution

Fraud detection systems relying on historical training data often struggle to detect new fraudulent modus operandi because emerging patterns lack labeled supervision. Grab developed GraphBEAN, an unsupervised autoencoder model designed to detect anomalous patterns across bipartite interaction graphs representing consumers and merchants. The architecture uses graph convolution layers to encode node and edge attributes, then reconstructs features and edge existence through separate feature and structure decoders. High reconstruction errors flag rare, anomalous behaviors at both the node and edge levels. An automated pipeline applies heuristic fraud tags to these scores and routes them to human analysts and automated mitigation systems.


### [What I learned taking Ramp Bill Pay from 0 to N](https://yomu.fyi/post/what-i-learned-taking-ramp-bill-pay-from-0-to-n.md)
- Company: [Ramp](https://yomu.fyi/company/ramp.md)
- Author: Pavel Asparouhov
- Published: Aug 1, 2023
- Type: Explainer

An engineer recounts lessons from taking Ramp’s Bill Pay from its first code in April 2021 to a publicly launched product that eventually became a product with multibillion annual total payment volume. For 0-to-1 work, the account emphasizes protecting core abstractions, including canonicalized payment statuses that reduce varied payment states to five, using mocked functions to coordinate parallel engineering, and reaching beta customers quickly. Customer feedback, session replay, and persistent iteration are presented as necessary because the first customer will usually expose substantial flaws. For 1-to-N growth, manual processes and duct tape must be prioritized for replacement according to user impact, error risk, and workload. Growing stakes also require testing, monitoring, contingency plans, gradual rollouts, QA, and documentation that helps engineers self-service onboarding.


### [Shopify’s platform is the Web platform](https://yomu.fyi/post/shopify-s-platform-is-the-web-platform.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Jul 26, 2023
- Type: Announcement

Shopify now recommends Remix for building Admin apps, which merchants use to configure apps, view data, and integrate services within Shopify Admin. Because these apps run in cross-origin iframes, browser restrictions can block cookies, storage, URL updates, and direct access to the Admin interface; App Bridge uses postMessage() to restore capabilities and support deeper integration. The new App Bridge client replaces the previous package, requires only a script tag, and makes standard web APIs such as history.pushState() work with Admin apps. The Remix template uses file-based routes, server-side loaders and actions, implicit API handling, and Shopify-specific integrations supplied by @shopify/shopify-app-remix, with Prisma provided for storage and sessions. Shopify integrated the template into Shopify CLI and describes it as the beginning of broader platform and design-system reworks.


### [Contributing support for a Wasm instruction to Winch](https://yomu.fyi/post/contributing-support-for-a-wasm-instruction-to-winch.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Jul 18, 2023
- Type: Tutorial

Adding a WebAssembly instruction to Winch, Wasmtime’s baseline non-optimizing compiler, requires coordinated changes across code generation, register management, ISA-specific emission, and tests. The contribution described adds visitor support, delegates through CodeGenContext and MacroAssembler, emits x86\_64 instructions through Cranelift-backed assembler methods, and registers the instruction for differential fuzzing. CodeGenContext keeps the value stack, register allocator, and function frame consistent, while spilling can move values to linear-memory offsets; MacroAssembler handles ISA differences, register constraints, and CPU feature fallbacks such as lzcnt versus bsr-based clz. Filetests inspect generated machine code without executing it, while differential fuzzing compares Winch results with another WebAssembly engine across randomly generated modules and inputs. The workflow also includes running fuzzing, incorporating draft-PR feedback, rebasing, and submitting the public Wasmtime pull request.


### [Moving Fast by Moving Slow: How We Built Payments at Ramp](https://yomu.fyi/post/moving-fast-by-moving-slow-how-we-built-payments-at-ramp.md)
- Company: [Ramp](https://yomu.fyi/company/ramp.md)
- Author: Shreyan Jain
- Published: Jul 17, 2023
- Type: Problem & solution

Ramp describes why it delayed the launch of BillPay by about two months to replace its existing, ad hoc bank-transfer infrastructure with a new payments service. The earlier system encoded product-specific concepts such as employee bank accounts and reimbursement transfers, forcing each new use case to add bespoke data models, jobs, and scheduling logic. This friction became especially costly as the team anticipated additional payment products and vendor payments scheduled for future dates. The redesign used generic primitives—ACHDetails and TransferCanonical—to separate business logic from money-movement logic; it also treats failed transfers and retries as concerns requiring an auditable identity, history, and consistent provider mapping. The post argues that stress-testing assumptions, modeling relevant degrees of freedom, limiting conditional logic, and building platforms after recurring patterns can make future products faster to launch.


### [Zero traffic cost for Kafka consumers](https://yomu.fyi/post/zero-traffic-cost-for-kafka-consumers.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Fabrice Harbulot
- Published: Jul 7, 2023
- Type: Problem & solution

Grab's real-time data streaming platform team reduced infrastructure expenses by enabling Kafka consumers to fetch from the closest partition replica instead of partition leaders. The platform previously operated across three AWS Availability Zones with three-way replication, resulting in cross-AZ network fees that constituted half of the Kafka platform's total cost. To address this, the team performed a zero-downtime upgrade to Kafka 3.1, configured a rack-aware replica selector using AWS Availability Zone IDs, and updated consumer SDKs to populate the client rack parameter. The rollout produced a 25% drop in cross-AZ traffic within three months under steady data volumes. However, the change increased end-to-end latency by up to 500ms, created potential CPU load skews across zones, and removed graceful isolation during broker rotation maintenance.


### [Playing Traffic Cop with Fly-Replay](https://yomu.fyi/post/playing-traffic-cop-with-fly-replay.md)
- Company: [Fly.io](https://yomu.fyi/company/fly-io.md)
- Published: May 15, 2023
- Type: Tutorial

Fly.io’s fly-replay response header lets the Fly Proxy replay an HTTP request on another app, region, or VM within the same organization. The post explains how this can send writes near a leader database, preserve request locality after uploads, wake Machines stopped on exit, or route customer traffic across apps. It then builds a small Go reverse-proxy application that looks up the request hostname in SQLite and returns either app=our-default-app or the registered customer’s app in the header. Local request handling and lookup took about 4 ms, while the real-world proxy-and-replay path added about 100 ms; private customer apps remained reachable because Fly services can be defined without public IP addresses. The post also mentions LiteFS and internal DNS as possible discovery mechanisms.


### [Next steps for Postgres pluggable storage](https://yomu.fyi/post/next-steps-for-postgres-pluggable-storage.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: paul\_copplestone
- Published: May 1, 2023
- Type: Explainer

PostgreSQL pluggable storage would let different tables in one database use storage engines optimized for transactional, analytical, or archival workloads, while allowing developers to build engines in an extensions-like model. PostgreSQL 12 added basic support through refactoring and expanding table access method APIs, initially to enable ZHeap, whose undo log showed promising results and addressed long-term table bloat, but work on ZHeap and TAM appears stalled. The current API has important gaps, including limited index flexibility, all-or-nothing index updates, and row-version identifiers tied to block and offset numbers; available options such as Citus Columnar Storage remain constrained by these limits. OrioleDB's upstreaming has reduced its fork's patchset from about 5,000 lines in PostgreSQL 14 to about 2,000 for PostgreSQL 16, raising the possibility that required changes could land in PostgreSQL 17.


### [Message Center - Redesigning the messaging experience on the Grab superapp](https://yomu.fyi/post/message-center-redesigning-the-messaging-experience-on-the-grab-supera.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Jonathan Lee
- Published: Apr 17, 2023
- Type: Problem & solution

Grab redesigned its messaging infrastructure from GrabChat to Message Center to overcome two-party chat limitations and support complex superapp requirements like group conversations and varied user roles. The architecture separates core processing logic from message delivery by splitting the system into a backend processor and an independently scalable postman service. Communication relies on an in-house TCP gateway named Hermes that proxies client payloads via gRPC, alongside Apache Kafka streams and Amazon SQS delay queues. Custom client-server acknowledgements and a DynamoDB event store ensure reliable message delivery even across dropped mobile TCP connections and offline reconnects.


### [Creating a Flexible Order Routing System with Shopify Functions](https://yomu.fyi/post/creating-a-flexible-order-routing-system-with-shopify-functions.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Apr 13, 2023
- Type: Explainer

Shopify’s smart order routing replaces a one-size-fits-all approach to choosing which inventory location fulfills an order with configurable routing rules. Rules can rank eligible locations by proximity to the destination, staying within the destination market, or merchant-defined location priorities, and merchants can arrange them in an order in the admin. Each rule is a Shopify Function written in Rust, receives data shaped through a GraphQL query, and returns location rankings; a reducer resolves ties by applying later rules to locations tied at rank 0. The rebuilt process uses the pipeline pattern to separate input preparation, rule execution, reduction, and final location selection, improving readability and debugging compared with the earlier tightly coupled implementation. This first stage dogfooded Shopify Functions on its own merchant stores, with a later stage planned to let developers add more complex rules.


### [Supabase Logs: open source logging server](https://yomu.fyi/post/supabase-logs-open-source-logging-server.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: chasers,ziinc
- Published: Apr 10, 2023
- Type: Announcement

Supabase is releasing Supabase Logs for self-hosted users and CLI development, built around Logflare, which processes more than 1 billion log events daily from Supabase services. Logflare is a multi-node, highly available Elixir cluster that ingests events through Vector or native integrations, streams them to BigQuery, and exposes SQL-backed HTTP Endpoints for Studio, Logs Explorer, and usage charts. Endpoints support parameterized queries, in-memory read-through caching, configurable cache warming, and select-only query sandboxing, allowing views over billions of daily events without querying many terabytes on every request. The release changes Logflare’s license to Apache 2.0, replaces closed-source SQL parsing with rust-based sqlparser-rs, and improves environment-based configuration for self-hosting; BigQuery is currently the supported backend, with more planned.


### [How Migrating from Vanilla Redux to Redux Toolkit Improved State Management in Shopify POS](https://yomu.fyi/post/how-migrating-from-vanilla-redux-to-redux-toolkit-improved-state-manag.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Mar 31, 2023
- Type: Explainer

Shopify’s React Native Point of Sale team migrated its production state management from Vanilla Redux to Redux Toolkit (RTK) after developers found boilerplate and debugging difficult. They rejected a full rewrite in favor of incremental reducer-by-reducer conversion, allowing Vanilla reducers and RTK slices to coexist and limiting rollback and destabilization risk. Most reducers adopted RTK’s mutable syntax and generated action creators, while asynchronous logic retained a hybrid thunk approach because RTK Query did not support GraphQL and wholesale use of pending and fulfilled actions would split logic between thunks and reducers. The roughly three-month migration deleted 3,500 lines of boilerplate and was released gradually to production. The team also identified explicit TypeScript typing for dispatch, selector, and store hooks, plus dispatch().unwrap() for awaited calls, as important details, while atypical reducers were deferred to separate projects.


### [What Being a Staff Developer Means at Shopify](https://yomu.fyi/post/what-being-a-staff-developer-means-at-shopify.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Mar 29, 2023
- Type: Explainer

The post defines a staff developer at Shopify as an individual contributor whose scope and seniority can match a manager’s, while emphasizing that the role is a form of leadership rather than simply a more senior senior developer. Technical excellence comes from varied, challenging project experience, including large-scale systems, refactors, technology shifts, mistakes, and learning to choose appropriate solutions under constraints. The role also pairs with a team manager, translating product and business objectives into technical proposals, judging scope and infrastructure needs, finding recurring product problems, and saying no when effort does not match return. Its impact is measured partly through enabling others via reviews, mentorship, sponsorship, and support, leading to an adaptable leader comfortable with ambiguity whose responsibilities evolve with company needs.


### [How OVO determined the right technology stack for their web-based projects](https://yomu.fyi/post/how-ovo-determined-the-right-technology-stack-for-their-web-based-proj.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: George Matthew Limongan
- Published: Mar 21, 2023
- Type: Problem & solution

OVO faced maintainability issues caused by fragmented web technology stacks, including PHP, Vue, React, Nuxt, and Go, alongside outdated documentation. This fragmentation resulted in severe context-switching costs during code reviews and prolonged onboarding times for new engineers. To establish a single primary front-end stack, the team compared React and Vue against maintainability principles encompassing operability, simplicity, and evolvability. Vue was selected because its standardized framework structure and documentation minimized architectural variations across projects. Implementation entailed creating a standardized Vue boilerplate, updating the internal UI library, and scheduling periodic upgrades for legacy React services rather than migrating them entirely.


### [Lessons From Linguistics: i18n Best Practices for Front-End Developers](https://yomu.fyi/post/lessons-from-linguistics-i18n-best-practices-for-front-end-developers.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Mar 16, 2023
- Type: Tutorial

Front-end internationalization can fail when code encodes English assumptions about word order, punctuation, pluralization, gender, or text layout. The post uses an image-status example to show why a complete sentence should be translated as a whole, then examines interpolation through a reusable status bar whose file-type and count values vary across languages. It explains that plural categories can include zero, one, two, few, many, and other, with Polish examples showing why numeric rules should not be hardcoded, and that gender can affect articles and adjectives, as in French. Its recommended practices include keeping linguistic decisions in translation strings, avoiding code-level text construction and elaborate styled layouts, and letting i18n libraries select plural forms from quantities; complex promo layouts may need simplification or language-specific images.


### [Migrating from Role to Attribute-based Access Control](https://yomu.fyi/post/migrating-from-role-to-attribute-based-access-control.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Minh Khoi Nguyen
- Published: Mar 9, 2023
- Type: Problem & solution

Grab's streaming data platform team migrated the Kafka Control Plane from Role-Based Access Control to Attribute-Based Access Control to eliminate operational bottlenecks and manual permission management. The previous model required defining hundreds of roles, permissions, and group mappings in an internal IAM service, leading to approval delays and stale memberships. Under the new architecture, user attributes sync from the HRMS and token payloads, while resource attributes are tagged upon creation or backfilled to reflect department and team ownership. Open Policy Agent evaluates access requests defined in Rego via middleware by comparing user attributes with resource metadata. This transition eliminated over 200 roles, 200 permissions, and roughly 3,000 unused IAM resources while automating access provisioning for new joiners.


[Newer posts](https://yomu.fyi/topic/architecture/page/30.md) · [Older posts](https://yomu.fyi/topic/architecture/page/32.md)