---
title: "Architecture"
description: "834 posts about Architecture, summarised, each linking to the original."
---

# Architecture
> 834 posts about Architecture, summarised, each linking to the original.

## Articles

### [How Grab built a scalable, high-performance ad server](https://yomu.fyi/post/how-grab-built-a-scalable-high-performance-ad-server.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Anthony McCallum
- Published: Feb 11, 2022
- Type: Problem & solution

Grab transitioned from an off-the-shelf MVP to an in-house ad serving system to accommodate business scale, hyperlocal requirements, and machine learning personalization. The architecture orchestrates core microservices and data pipelines across sequential steps: targeting, capping, pacing, scoring, ranking, pricing, and tracking. ElasticSearch serves as the targeting ads repository, while ScyllaDB acts as the high-throughput stats store fed by Kafka streams and data pipelines. The system operates on key engineering principles including parallelization and tuned latency limits, graceful fallbacks for slow dependency calls, and a unified server serving all ad types across the superapp.


### [Hydrogen & Tailwind: The Perfect Match for Building Beautiful Storefronts](https://yomu.fyi/post/hydrogen-tailwind-the-perfect-match-for-building-beautiful-storefronts.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Feb 11, 2022
- Type: Explainer

Hydrogen uses Tailwind as the default styling option in its React-based starter, allowing developers to build storefronts with utility classes from the moment they run create-hydrogen-app. The article presents componentization as a central way to organize those classes: React components can encapsulate responsive, hover, flexbox, spacing, typography, and visibility utilities and be reused across a storefront. It contrasts this with semantic class systems whose names can become coupled to a product component, making reuse, renaming, or duplication more cumbersome. Tailwind’s learning curve still requires CSS knowledge, but its documentation and VS Code extension support autocomplete, style representations, and color previews. The conclusion is that Tailwind’s constraints, customizable design system, and composable styles complement Hydrogen’s starter components and developer experience, although the author acknowledges that readers may still prefer other approaches.


### [Shopify's Playbook for Scaling Machine Learning](https://yomu.fyi/post/shopify-s-playbook-for-scaling-machine-learning.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Feb 11, 2022
- Type: Explainer

Shopify describes how its machine-learning practice grew from a first order-fraud product into a portfolio used across products such as Shopify Capital, product categorization, and Help Center search. The playbook starts with choosing a user-important problem, validating trustworthy and accessible data, tracing downstream dependencies, understanding existing solutions, and optimizing for product outcomes rather than model scores alone. To productionize models, Shopify recommends well-defined training, verification, and testing pipelines, deployment decisions based on volume and user commitment, and platform practices that encode validated standards without abstracting too early. Its fraud pipeline builds models in Python, serializes them with PMML for deployment in Ruby-based production systems, and uses Apache Airflow to schedule operations. The account concludes that scaling is a full-stack problem requiring data scientists and data engineers to align on one roadmap and goal.


### [React Server Components Best Practices You Can Use with Hydrogen](https://yomu.fyi/post/react-server-components-best-practices-you-can-use-with-hydrogen.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Feb 4, 2022
- Type: Tutorial

React Server Components (RSC) introduced a paradigm shift for the team building Hydrogen, a React-based framework for custom storefronts. The post presents patterns developed through trial and error, beginning with shared components as a middle ground and reserving client components for interactivity, useState or useReducer, lifecycle logic, unsupported libraries, or browser APIs. Components should instead become server components when they contain secrets or proprietary logic, need filesystem or database access, fetch storefront API data, or do not execute on the client. The examples extract newsletter form controls and FAQ accordion behavior into client components, then rename the remaining wrappers as server components when their usage allows it; the FAQ server component is passed through a client component as children.


### [Rapid Development with Hydrogen: Building a Product Page](https://yomu.fyi/post/rapid-development-with-hydrogen-building-a-product-page.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Feb 2, 2022
- Type: Tutorial

Updated for Hydrogen 0.26.0, the tutorial builds a simple product page for a custom Shopify storefront using Hydrogen’s React-based framework. Starting from the Hello World template on StackBlitz, it adds Tailwind CSS, creates a file-based dynamic product route, and queries product data through the Storefront API with the useShopQuery hook. The implementation then uses React Server Components alongside a client component and ProductOptionsProvider to track selected variants and options, update prices, and support a BuyNowButton checkout flow. A media gallery and additional styling complete the page, which uses products and media from the Hydrogen Preview store. The post concludes that Hydrogen’s components, hooks, and utilities reduce repetitive storefront work and speed development.


### [Biometric authentication - Why do we need it?](https://yomu.fyi/post/biometric-authentication-why-do-we-need-it.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Chad Burgess
- Published: Jan 20, 2022
- Type: Problem & solution

Grab addressed the vulnerabilities and costs associated with SMS one-time passwords and PINs by implementing device-level biometric authentication. The architecture pairs device biometric sensors with hardware secure enclaves to protect private keys separately from the main operating system. During enrollment, Grab generates a public-private key pair using SHA512withECDSA, authenticates the user locally, and stores reference identifiers in encrypted device storage. HellfireSDK verifies that the device is not rooted, ensuring raw biometric data never leaves the handset. Early experimental runs indicate an adoption rate exceeding 90% and a login success rate near 90%.


### [Cloud, Load, and Modular Code: What 2022 Looks Like for Shopify](https://yomu.fyi/post/cloud-load-and-modular-code-what-2022-looks-like-for-shopify.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Jan 20, 2022
- Type: Announcement

Shopify’s 2021 BFCM four-day event exposed the scale its commerce platform must absorb: peak app-server traffic exceeded 32 million requests per minute, load balancers surpassed 34 million RPM, and a flash sale used more than 20% of total computing capacity. The post frames larger, spikier sales, mobile and social commerce, and expanding buying environments as enduring engineering challenges for supporting millions of merchants. Planned work includes evolving the large Rails codebase into modular components, moving storefront rendering out of the monolith, expanding Spin cloud development environments, and making in-house load testing part of product development. The storefront split and refactoring produced four-times-faster cache fill rates and five-times-faster page rendering, while YJIT ran 31% faster than interpreted CRuby and 26% faster than MJIT; Shopify also remains committed to Rails and React Native.


### [Shopify’s Unique Data Science Hierarchy Of Needs](https://yomu.fyi/post/shopify-s-unique-data-science-hierarchy-of-needs.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Dec 21, 2021
- Type: Explainer

Shopify presents a five-level Data Science Hierarchy of Needs that organizes data work by the value it delivers, rather than by technical sophistication. Its tool-agnostic philosophy favors simple solutions first, then more complex analysis when necessary, with impact and better decisions as the goal. The model begins with collecting and modeling clean, conformed data using dimensional modeling, then moves through description, prediction and inference, prescription, and influence. Shopify applied this structure to a COVID-19 impact analysis, using statistical analysis to examine regional effects and inform measures including an extended free trial, expanded funding, shipping options, and local commerce features. The account says the analysis used neither artificial intelligence nor machine learning, yet supported decisions for merchants and Shopify during the pandemic.


### [Perceiver IO: a scalable, fully-attentional model that works on any modality](https://yomu.fyi/post/perceiver-io-a-scalable-fully-attentional-model-that-works-on-any-moda.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Niels Rogge
- Published: Dec 15, 2021
- Type: Announcement

Standard Transformer architectures scale poorly in compute and memory because pairwise dot-product self-attention depends quadratically on input size. Perceiver IO addresses this constraint by computing self-attention across a small set of latent variables rather than directly on high-dimensional inputs. Inputs and outputs interact with the model via cross-attention operations, decoupling compute and memory costs from input and output dimensions. Integrated into Hugging Face Transformers via the PerceiverModel class, the architecture supports diverse data types using optional preprocessors, decoders, and postprocessors. Experiments demonstrate competitive performance across text, multimodal video classification, 3D point cloud classification on ModelNet40, and StarCraft II reinforcement learning in AlphaStar.


### [Building a Real-time Buyer Signal Data Pipeline for Shopify Inbox](https://yomu.fyi/post/building-a-real-time-buyer-signal-data-pipeline-for-shopify-inbox.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Dec 13, 2021
- Type: Problem & solution

Shopify built a real-time buyer-signal pipeline for Shopify Inbox after noting that 70 percent of its conversations involved customers making a purchasing decision. The system surfaces cart actions and order-completion information as chronological conversation events, covering events from up to 14 days before a conversation and post-conversation events while its seven-day lifecycle remains active. It uses Kafka through Monorail and Change Data Capture, Apache Beam on Google Cloud Dataflow, filtering jobs for mission-critical transactional events, and stateful aggregation with timers and a Global Window to retain buyer activity. In an equal-group A/B test, buyer signals increased merchant response rate by two percentage points and attributed conversion rate by 0.7 percentage points, while response time showed no significant change.


### [Realtime Postgres RLS now available on Supabase](https://yomu.fyi/post/realtime-postgres-rls-now-available-on-supabase.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: Oliver Rice
- Published: Dec 1, 2021
- Type: Announcement

Supabase updated its Realtime server to enforce PostgreSQL Row Level Security (RLS) policies when broadcasting database changes over websockets. Previously, Realtime operated as an opt-in beta feature that sent all replication changes to every client regardless of user authorization. To enforce RLS per subscriber without heavy performance overhead, Supabase introduced WALRUS, a security engine colocated inside PostgreSQL. For each replication change, WALRUS looks up active subscribers, assumes their identities, and evaluates row visibility using prepared statements queried by primary key. This in-database evaluation avoids external network round trips and single-query planning overhead while returning an authorized subscriber list to Realtime.


### [Remote rendering: Shopify’s take on extensible UI](https://yomu.fyi/post/remote-rendering-shopify-s-take-on-extensible-ui.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Dec 1, 2021
- Type: Explainer

Shopify describes a UI extension architecture designed to give third-party code consistent, accessible, cross-platform experiences without allowing it to compromise the host application. Its remote-rendering approach separates UI definition from rendering, with extensions communicating with hosts through MessageChannel-based message passing; hosts are implemented in JavaScript, Kotlin, and Swift. The open-source remote-ui library supplies RPC for passing callbacks across postMessage, automatic memory management, a DOM-like RemoteRoot that serializes updates as JSON, and receivers that map remote components to DOM or React components. A sandbox isolates untrusted extension code and controls exposed APIs across web workers, Android web views, and iOS JsCore. The system powers product-subscription and post-purchase extensions, while a DOM-based version began releasing in July 2025.


### [Building an App Clip with React Native](https://yomu.fyi/post/building-an-app-clip-with-react-native.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Nov 26, 2021
- Type: Problem & solution

The Shop team built an iOS App Clip to reduce the drop-off between a buyer’s checkout thank-you page and opening the full Shop app, where App Store download and onboarding added friction. App Clips are invoked through actions such as QR scans, links, or Smart App Banners and must stay below 10MB, while the team wanted React Native support for order tracking and related functionality. A Hello World build initially measured 28MB; explicitly selecting native dependencies in the Podfile reduced the baseline to 4.3MB, and a scrappy feature build reached 6.5MB. The team used early testing, invocation URL handling through React Native Linking, and shared data containers for app upgrades, ultimately releasing an App Clip at 9.1MB; extensive code sharing proved too complex and some code was copied.


### [Using real-world patterns to improve matching in theory and practice](https://yomu.fyi/post/using-real-world-patterns-to-improve-matching-in-theory-and-practice.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Tenindra Abeywickrama
- Published: Nov 22, 2021
- Type: Problem & solution

Continuous ride-hailing assignment relies on solving the minimum weight bipartite matching problem between passengers and driver-partners. While traditional implementations assume a precalculated cost matrix, computing shortest-path travel times across large road networks dominates total execution time. Researchers introduced an Incremental Kuhn-Munkres algorithm that leverages the spatial locality of optimal matches to compute edge costs on demand. The approach integrates priority queues and lower-bounding techniques with refinement rules to avoid evaluating distant pairs while guaranteeing the same optimal assignment. Evaluated on Singapore road network data and real Grab production workloads, the incremental techniques reduced exact cost calculations and decreased assignment running times by over an order of magnitude.


### [Bootstrapping a UI component library](https://yomu.fyi/post/bootstrapping-a-ui-component-library.md)
- Company: [Ramp](https://yomu.fyi/company/ramp.md)
- Author: Nicolás Bevacqua
- Published: Nov 15, 2021
- Type: Explainer

Ryu is a component library created at a small startup to separate design concerns from feature development and business logic while product and conventions continue changing. It mirrors Figma's design system through shared colors, sizes, fonts, line heights, media breakpoints, z-indices, and components such as inputs, buttons, tooltips, menus, tables, and drawers. The article argues for a hard boundary between the library and main codebase, using a separate repository and an npm package with documented interfaces, hand-crafted changelogs, versioned releases, and extensive TypeScript. It emphasizes resisting narrowly tailored settings, postponing unresolved use cases, and prioritizing adoption over completeness, including preventing new code from using legacy components without good reason. Building on mature existing components while simplifying their interfaces, the library aims to improve maintainability, consistency, accessibility, theming, contribution, and coordination across applications.


### [Building Blocks of High Performance Hydrogen-powered Storefronts](https://yomu.fyi/post/building-blocks-of-high-performance-hydrogen-powered-storefronts.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Nov 8, 2021
- Type: Explainer

Hydrogen is a React-based framework for custom Shopify storefronts, designed to balance fast loading, personalized and dynamic commerce, and developer experience. The central trade-off is that static generation and edge delivery can speed loading but defer personalization, while conventional server-side rendering can slow time-to-first-byte as data is fetched. Hydrogen addresses this with React 18 alpha streaming SSR powered by Suspense, which streams the page shell before data resolves and progressively hydrates components without extra client round trips. React Server Components keep server-only code out of bundles, support server-side data access, and enable component-level updates that preserve client state. Built-in fetching and configurable caching, including stale-while-revalidate, work with Oxygen’s colocated runtime, while developers can mix server rendering, client fetching, and edge delivery at page or component level.


### [The Age of Machine Learning As Code Has Arrived](https://yomu.fyi/post/the-age-of-machine-learning-as-code-has-arrived.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Julien Simon
- Published: Oct 20, 2021
- Type: Explainer

Recent findings from the 2021 State of AI Report and Kaggle State of Machine Learning and Data Science Survey indicate that machine learning is expanding into critical infrastructure while Transformers become general-purpose architectures across text, vision, and audio. In response, organizations face questions about scaling infrastructure, team composition, and engineering maturity. Rather than treating machine learning as isolated sandbox experiments or hiring solely data scientists, teams benefit from adopting established software engineering and DevOps principles like versioning, testing, automation, and continuous deployment. Furthermore, the rise of pre-trained Transformer architectures enables practitioners to fine-tune existing off-the-shelf models rather than training from scratch, reducing compute costs and training duration. Tools from platforms such as Hugging Face streamline model deployment, latency optimization, and infrastructure abstraction.


### [YJIT: Building a New JIT Compiler for CRuby](https://yomu.fyi/post/yjit-building-a-new-jit-compiler-for-cruby.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Oct 14, 2021
- Type: Explainer

YJIT is a new just-in-time compiler being built inside CRuby to improve Ruby performance on modern hardware, where single-core speed gains are limited and energy efficiency matters. Unlike MJIT’s weaker results on large real-world applications, YJIT uses a data-driven focus on performance hotspots in Rails and Shopify Core and is based on Basic Block Versioning (BBV). After about one year, it reports speedups over the CRuby interpreter of 20% on railsbench, 39% on liquid template rendering, and 37% on activerecord, with near-peak performance after one benchmark iteration. YJIT passes CRuby’s roughly 30,000-test suite, Shopify Core and GitHub backend CI, while running on a small percentage of Shopify production servers; its team plans further CRuby changes and broader JIT coverage before and after its intended Ruby 3.1 integration.


### [Reusing Code with React Native Packages at Shopify](https://yomu.fyi/post/reusing-code-with-react-native-packages-at-shopify.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Sep 29, 2021
- Type: Explainer

Shopify’s React Native teams needed to reuse shared functionality across Shop, Inbox, Point of Sale, Shopify Mobile, and Local Delivery without requiring each team to build and maintain separate package infrastructure. The React Native Foundations team chose a monorepo containing npm packages, a fixture app, shared root configuration, generated CI, and an internal documentation site, with code generation for package bootstrapping. Packages use TypeScript project references, Yarn Workspaces, Lerna publishing, and custom Babel, Metro, and Podfile configuration to support cross-package references, hot reload, and local native dependencies. Dependency graphs and file-change triggers limit CI work to affected packages and dependents, while independent versioning and release scripts manage publication; since June 2020, 17 packages have been developed, including 10 contributed by product teams, although release automation and coordination across multiple monorepos remain future improvements.


### [Shard Balancing: Moving Shops Confidently with Zero-Downtime at Terabyte-scale](https://yomu.fyi/post/shard-balancing-moving-shops-confidently-with-zero-downtime-at-terabyt.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Sep 24, 2021
- Type: Explainer

Shopify’s MySQL shards can become unevenly utilized as merchants grow after being assigned to arbitrary shards, increasing over-utilization risk while leaving other capacity underused. The post describes a strategy for selecting shop placements using historical utilization and traffic data, simulations, and potentially other metrics such as shop size, GMV, and move time. It then explains online shop migration with Ghostferry, which batch-copies records and tails MySQL’s binary log before a brief cutover updates the routing table and directs new writes to the target pod. Verification checks data completeness, integrity, and stale traffic before old data is pruned; the migration system’s central algorithm is also represented in a TLA+ formal specification. The stated outcome is a more balanced infrastructure that can isolate high-traffic merchants across pods with virtually zero consumer-facing downtime.


[Newer posts](https://yomu.fyi/topic/architecture/page/35.md) · [Older posts](https://yomu.fyi/topic/architecture/page/37.md)