---
title: "Architecture"
description: "834 posts about Architecture, summarised, each linking to the original."
---

# Architecture
> 834 posts about Architecture, summarised, each linking to the original.

## Articles

### [Mastering React’s Stable Values](https://yomu.fyi/post/mastering-react-s-stable-values.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Aug 2, 2022
- Type: Explainer

React stable values are values, often returned by hooks, that retain the same value across renders, and the post explains why their identity matters in effects, callbacks, memoization, and component rendering. useState and useReducer update functions and useRef objects are stable, while array literals, object literals, and anonymous functions are recreated on each render; useCallback and useMemo can produce stable values under appropriate dependencies. Unstable dependencies prevent useEffect behavior from remaining limited to intended changes and stop useCallback and useMemo cached results from being reused, while an unstable callback passed to useFocusEffect causes repeated unsubscribe and resubscribe cycles. The post also contrasts invoking an inline helper with rendering it as a component, explains why the latter can rebuild a React Native subtree on every render, and recommends defining components outside other components and using React.memo or callbacks judiciously.


### [Implementing "seen by" functionality with Postgres](https://yomu.fyi/post/implementing-seen-by-functionality-with-postgres.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: Victor
- Published: Jul 18, 2022
- Type: Benchmark

Tracking unique post views within Postgres presents trade-offs between counter accuracy, row bloat, and concurrent write performance. To evaluate options under an architecture constraint preventing external dependencies, a benchmark suite generated synthetic users, skewed post distributions, and replayed view actions. The benchmark tested four implementations: a naive counter column, an hstore key-value approach, an association table, and HyperLogLog (HLL). Benchmark results demonstrated that simple-hstore achieved the lowest average latency among deduplicating approaches at 2.15 milliseconds, closely followed by HLL at 2.16 milliseconds. Despite hstore's raw performance in prototypes, HLL is recommended for production because it circumvents row bloat as view counts grow while avoiding expensive row counts.


### [Typeahead Search at Nextdoor](https://yomu.fyi/post/typeahead-search-at-nextdoor.md)
- Company: [Nextdoor](https://yomu.fyi/company/nextdoor.md)
- Author: Jerry Tian
- Published: Jul 6, 2022
- Type: Problem & solution

Nextdoor built a proximity-based autocomplete service to power typeahead search and mention features across its hyperlocal platform for hundreds of millions of entities, including users and businesses. The system shards geographic data using Uber's open-source H3 geohashing library and stores prefix indexes in memory using Redis sorted sets. By adopting a Command Query Responsibility Segregation architecture, ingestion writes are processed on Redis primary nodes and replicated to read-only search nodes with under 10 milliseconds of replication lag. Dedicated APIs handle indexing, typeahead lookups, and ranking before returning hydrated results. Operating since August 2021, the service processes hundreds of millions of monthly typeahead queries while maintaining a P95 search latency below 30 milliseconds.


### [Shopify and Open Source: A Mutually Beneficial Relationship](https://yomu.fyi/post/shopify-and-open-source-a-mutually-beneficial-relationship.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Jul 6, 2022
- Type: Explainer

Shopify describes its relationship with Ruby, Rails, and the open-source community as mutually beneficial, linking the company’s success to its Rails foundation and its engineering culture to the Rails Doctrine. It frames a 100-year company ambition around making Ruby and Rails viable for large teams, mature codebases, millions of lines of code, and billions of web requests through production performance, architecture guidance, upgrades, tooling, and selective typing. To support that goal, Shopify contributes projects and initiatives spanning developer tooling, gradual typing, VS Code support, automated upgrades, YJIT, Ruby performance work, TruffleRuby collaboration, $500,000 in research funding, Bundler and RubyGems, and RubyGems.org security. The post argues that company participation improves engineering skills, application longevity, and influence over foundational software, while describing contributors who range from occasional participants to full-time maintainers.


### [The Story Behind Shopify’s Isospin Tooling](https://yomu.fyi/post/the-story-behind-shopify-s-isospin-tooling.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Jul 5, 2022
- Type: Explainer

Shopify’s Isospin is the systemd-based core of Spin, its in-house cloud development platform, designed to run multiple applications inside a single Linux VM. The earlier implementation relied on shell scripts, but growing application counts made it difficult to split applications into components, express dependencies, schedule jobs, and isolate processes. Isospin uses systemd unit graphs, targets, template units, and generators to create service configurations dynamically, including per-application copies of dependencies such as MySQL and Elasticsearch. Additional tooling assigns semi-stable ports by hashing service names and uses systemd readiness notifications, including a notify-port wrapper that checks HTTP availability; the resulting system reuses systemd’s dependency and health mechanisms, with more complex cross-application relationships under investigation.


### [Graph Networks - 10X investigation with Graph Visualisations](https://yomu.fyi/post/graph-networks-10x-investigation-with-graph-visualisations.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Fujiao Liu
- Published: Jun 30, 2022
- Type: Problem & solution

Fraud detection traditionally required investigators to manually combine large datasets from disparate anti-fraud systems using statistical methods, which proved slow and inefficient. Grab built an interactive Graph Visualisation platform to transform raw records into connected visual maps without requiring manual queries or switching tools. The platform manages over three billion nodes and edges, allowing investigators to selectively expand data points and replay chronological events using temporal filters. Visual relationship mapping helps teams verify account appeals, uncover device-sharing rings, and spot anti-money laundering behavior through transaction density patterns.


### [The Serverless Server](https://yomu.fyi/post/the-serverless-server.md)
- Company: [Fly.io](https://yomu.fyi/company/fly-io.md)
- Published: Jun 30, 2022
- Type: Explainer

The post explains what “serverless” means by reconstructing the machinery beneath a Lambda-like platform. It starts with bare-metal x86 servers, a Type 1 hypervisor, and services for placement, VM management, request forwarding, and function execution; dedicated customer workers, shared warm pools, rate limits, and container-style isolation address multitenancy, cold starts, and bursty demand. The runtime design uses object storage for function code, chroot, privilege dropping, namespaces, seccomp, cgroups, and freezing to constrain idle instances and workloads. Its limits on initialization, invocation duration, and idle lifecycles support migration away from unhealthy instances but disadvantage applications with expensive startup, dynamic local caches, or uncertain response times.


### [Navigating Recurring Payments in India: A Backend Perspective](https://yomu.fyi/post/navigating-recurring-payments-in-india-a-backend-perspective.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Jun 29, 2022
- Type: Explainer

India’s October 2021 Reserve Bank of India (RBI) directive required recurring payments to use payer-authorized e-mandates, Additional Factor of Authentication (AFA), and pre-debit notifications at least 24 hours before charges. Shopify’s billing platform therefore had to model e-mandates, support recurring and one-time payments across cards and Unified Payments Interface (UPI), and integrate a local payment provider. The payment service reused its source object for the e-mandate token, recorded the temporary authorization as a charge for auditability, and refunded it asynchronously. To avoid two consecutive card authentication flows, the team used recurring-payment tokenization for both payment paths, accepting an atypical notification experience as a short-term trade-off; UPI required asynchronous webhook-driven handling and a hard mandate limit. After the beta launch, billing-payment success rates in India increased from pre-project levels, and the team said it would soon make UPI generally available to Indian merchants.


### [How We Built Shopify Party](https://yomu.fyi/post/how-we-built-shopify-party.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Jun 14, 2022
- Type: Explainer

Shopify Party is a browser-based internal tool designed to make virtual hangouts more social and reduce video fatigue after Shopify moved to remote work. Its 3D spaces favor non-prescriptive play, letting participants explore and manipulate objects, join or leave at any point, and use a small control set rather than follow prescribed rules or win conditions. The initial version was built in three months with Unity 2020 and the Universal Render Pipeline, exported to WebGL, while Normcore synchronizes game state, provides VoIP and networked physics, and uses room time to keep autonomous objects aligned across clients. Performance work included third-party assets, GPU instancing for repeated models, simpler materials, and adjustable render resolution. The launch exceeded expectations, with hundreds of games played each week, and the tool is positioned as an option alongside video conferencing rather than a replacement.


### [Director of Machine Learning Insights \[Part 3: Finance Edition\]](https://yomu.fyi/post/director-of-machine-learning-insights-part-3-finance-edition.md)
- Company: [Hugging Face](https://yomu.fyi/company/hugging-face.md)
- Author: Britney Muller
- Published: Jun 14, 2022
- Type: Explainer

Machine learning leaders in financial institutions face significant operational hurdles when integrating automated models into production environments. Practitioners must navigate complex legacy architectures, strict regulatory oversight, and privacy mandates tied to personally identifiable financial records. Integrating models often fails when organizations lack clear communication buffers, deploy models using improper prediction windows, or treat systems as opaque black boxes without verifying underlying mechanics. Despite these challenges, financial applications increasingly rely on machine learning for anti-money laundering compliance, automated fraud screening, and loan underwriting. Success in this regulated sector demands comprehensive evaluation across representative input spaces, robust ongoing model monitoring, and explainable decision outputs.


### [How facial recognition technology keeps you safe](https://yomu.fyi/post/how-facial-recognition-technology-keeps-you-safe.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Kai Feng Tee
- Published: Jun 9, 2022
- Type: Explainer

Grab utilizes facial recognition technology across its platform for driver authentication, passenger verification, and digital electronic Know Your Customer (e-KYC) processes. The core pipeline consists of image preprocessing through face detection and alignment, anti-spoofing checks, feature extraction into high-dimensional vector embeddings, and downstream verification or search. To counter spoof attacks like screen replays, synthetic moiré patterns are generated and cropped face patches are used during training and inference to focus on local structures rather than global semantic noise. Face verification challenges involving shallow ID datasets and masked faces are resolved using semi-Siamese training architectures and masked data augmentations.


### [The Journey to Cloud Development: How Shopify Went All-in on Spin](https://yomu.fyi/post/the-journey-to-cloud-development-how-shopify-went-all-in-on-spin.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Jun 9, 2022
- Type: Problem & solution

Shopify’s development environment, the Ruby-based dev tool and xhyve-backed railgun VM, no longer handled the growing number of repositories, services, integrations, and resource demands around its monoliths. Developers needed more services, faced sluggish code-build-test loops, and encountered costly state changes when tophatting collaborators’ work. The Environments team concluded that development environments needed cloud-like elasticity, then tested an automatically configured local Kubernetes cluster and easy creation of Google Compute Engine VMs; the source also describes Isospin and Spin as the resulting direction. Local Kubernetes did not gain broad organic traction, while the current Isospin implementation is described as the team’s best iteration for developers working on majestic monoliths. The team plans to evolve it into an application-development library built on dev, add dependency inference, and address the tension between persistent environments and Kubernetes pod relocation.


### [Graph concepts and applications](https://yomu.fyi/post/graph-concepts-and-applications.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Wenxiang Lu
- Published: Jun 2, 2022
- Type: Explainer

Real-world systems generate dynamic, non-random connections that traditional statistical approaches fail to characterize or forecast. Graph models represent these structures through vertices and edges, abstracting complex networks into mathematically tractable relationships. Common data representation formats include the Resource Description Framework (RDF), which models subject-predicate-object triples with IRIs, literals, and blank nodes, and Labeled Property Graphs (LPGs), which store arbitrary key-value properties directly on nodes and edges. Graph databases, derived from the LPG model, treat relationships with equal weight to entities, delivering responsive traversals for highly interconnected systems. While they offer agility and explicit relationship modeling, graph databases lack a standardized query language and remain poorly suited for standard transaction-focused workloads compared to relational databases.


### [To Thread or Not to Thread: An In-Depth Look at Ruby’s Execution Models](https://yomu.fyi/post/to-thread-or-not-to-thread-an-in-depth-look-at-ruby-s-execution-models.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: May 31, 2022
- Type: Explainer

Threaded servers such as Puma and thread-based Sidekiq are popular Ruby deployment choices because threads can increase throughput without a proportional increase in memory. The post compares process and thread models using a memory model that separates static application memory from per-request processing memory, then refines it with Copy on Write (CoW), where forked processes share pages until writes force copies. It explains why RSS can double-count shared pages, recommends PSS and Linux’s /proc/$PID/smaps\_rollup for more meaningful measurements, and gives a production example in which about two-thirds of static memory is shared. The conclusion is metric-dependent: preload applications to improve CoW, prefer fewer larger containers when appropriate, consider Unicorn for reliable request timeouts, and start web workloads at two threads unless substantial I/O wait justifies more; job processors may benefit from higher counts.


### [Automated Experiment Analysis - Making experimental analysis scalable](https://yomu.fyi/post/automated-experiment-analysis-making-experimental-analysis-scalable.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Albert Cheng
- Published: May 30, 2022
- Type: Problem & solution

Manual ad-hoc analysis of online controlled experiments at Grab introduced operational inefficiencies, inconsistent quality control, and scalability barriers across teams. To resolve these issues, Grab extended its GrabX experimentation platform with an Automated Experiment Analysis system that standardises metrics and automates statistical evaluations. The architecture stores experiment configurations and metric definitions from Cosmos DB into Azure Data Lake as bronze datasets, uses Spark on Databricks via Azure Data Factory to process subjects into silver datasets, and applies an internal Python Decision Engine to generate final gold results. These gold datasets are stored in star-schema fact and dimension tables and presented directly in the GrabX interface using embedded Power BI visualisations. The automation eliminates repetitive data pipeline construction for analysts, ensures reproducible findings aligned with initial hypotheses, and accelerates product launch decisions.


### [Our Favorite Moments From RailsConf 2022](https://yomu.fyi/post/our-favorite-moments-from-railsconf-2022.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: May 24, 2022
- Type: Explainer

RailsConf 2022 returned to an in-person format at Portland’s Oregon Convention Center after a two-year gap, with three days of keynotes, tutorials, and talks on technical and workplace topics. The 16th edition announced three new Rails Core team members, including Shopify’s Jean Boussier, and two new committers, while recurring themes included company investment in Rails, security, Active Records, “build vs buy,” diversity, and burnout. Shopify’s 15 Rails speakers contributed sessions and keynotes, and the recap gathers highlights on open-source staffing, long-term maintenance, developer wellbeing, Rails upgrades, supply-chain attacks, and modularization for “Big Rails.” Specific examples included regexp DOS risks, typosquatting and combosquatting, Rails upgrade automation, Ruby Archaeology, and the history of women in computer science. Because concurrent sessions and hallway conversations limited attendance, a virtual home edition and recordings were scheduled to make more talks available afterward.


### [Double Entry Transition Tables: How We Track State Changes At Shopify](https://yomu.fyi/post/double-entry-transition-tables-how-we-track-state-changes-at-shopify.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: May 18, 2022
- Type: Explainer

Shopify needed a reliable way to count merchants using Shopify Balance over time, where usage requires both an active Shopify Balance account and an active Shopify account. The data team built accounts\_transition\_facts with double entry transition tables, a format that represents each state change as a -1 row for the previous state and a +1 row for the new state. Individual attribute tables use account\_id, transition\_at, and an index that breaks timestamp ties; the pipeline merges attributes, carries forward missing values, and generates net\_change rows. Because net\_change is additive, queries can sum it while filtering statuses, group results by date, or feed daily aggregates, while additional attributes can be added without rewriting existing SQL or PySpark. The initial build required substantial effort but produced a reporting structure intended to scale with Shopify Balance’s growing complexity.


### [Search architecture revamp](https://yomu.fyi/post/search-architecture-revamp.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Lipeng Zhang
- Published: May 17, 2022
- Type: Problem & solution

Before 2021, Grab’s Deliveries search architecture used an inverted index for textual matching, leaving it unable to account for user intention, context, or non-textual recall methods. Queries such as “Roti Prata” could prioritize merchant names over dishes, while the same query could imply different objectives by time or location; merging Ads and organic results at the Deliveries API layer also discarded recall-stage data. The revamped flow adds boosting, multi-recall, and ranking, enabling intent recognition, query expansion, embedding-based and graph-based recall, and joint ranking across business verticals. A platform-defined framework assembles service-team plugins through configuration, separating core flow from application logic and allowing components such as Ads and Mart recall to be reused across Food, Mart, and universal search flows. The resulting platform provides intent recognition, semantic search, mixed Ads-and-organic ranking, and simpler onboarding, while making new search modules and use cases easier to add.


### [Maestro: The Orchestration Language Powering Shopify Flow](https://yomu.fyi/post/maestro-the-orchestration-language-powering-shopify-flow.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: May 13, 2022
- Type: Explainer

Shopify developed Maestro, a domain-specific orchestration language, to decouple Shopify Flow’s merchant-facing visual editor from its execution engine, whose requirements differ in declarative usability versus scalable, fault-tolerant execution. The team introduced a horizontally scalable Maestro engine, translated existing Flow workflows into Maestro, migrated them incrementally, and later connected a new visual language directly to Maestro; all workflows had moved to the new editor and engine by early 2022. Maestro coordinates calls to host-language functions rather than implementing service or database operations, while durable checkpoints capture completed work and let the interpreter recover after crashes with at-least-once semantics for primitive calls. During activation, static usage analysis identifies needed returned attributes for possible query optimization, and compilation removes runtime-irrelevant definitions before deployment; checkpoints also support execution monitoring, while planned work includes richer language features, orchestration merging, and workflow preview and testing.


### [Our Experience Porting the YJIT Ruby Compiler to Rust](https://yomu.fyi/post/our-experience-porting-the-yjit-ruby-compiler-to-rust.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: May 11, 2022
- Type: Explainer

Shopify’s YJIT team moved the CRuby JIT compiler from C99 to Rust after finding that C’s limited complexity-management tools made the growing project difficult to maintain. The port had to preserve a non-trivial interface with CRuby, including parsing bytecode and manipulating Ruby’s primitive types, while compiling the Rust project into a static library linked through CRuby’s makefile. The team translated functions and structs largely one by one, retaining the original architecture while using Rust features such as pattern matching, macros, Cargo features, and borrow checking. Completed in three months, the port was judged more maintainable and pleasant to work with, although the team identified challenges around C interoperability, unsafe code, tooling, ecosystem maturity, and limited use of idiomatic Rust.


[Newer posts](https://yomu.fyi/topic/architecture/page/33.md) · [Older posts](https://yomu.fyi/topic/architecture/page/35.md)