---
title: "Architecture"
description: "834 posts about Architecture, summarised, each linking to the original."
---

# Architecture
> 834 posts about Architecture, summarised, each linking to the original.

## Articles

### [Making Shopify’s Flagship App 20% Faster in 6 Weeks Using a Novel Caching Solution](https://yomu.fyi/post/making-shopify-s-flagship-app-20-faster-in-6-weeks-using-a-novel-cachi.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Sep 22, 2021
- Type: Problem & solution

Shop’s home feed, the app’s most used feature, aggregated orders from millions of Shopify and non-Shopify merchants plus tracking data from dozens of carriers, accounted for 30% of database load and affected application performance. Before each database update, the custom write-through cache marks a user’s cache invalid, applies the write, then refreshes and revalidates it. To handle concurrent updates, it uses a separate pending-writes key and a short-expiry key to distinguish active writes from interrupted processes, while Active Record Concerns preserve the existing ORM interface. After a staged validation rollout, the cache reduced database load by 15%, overall app latency by about 20%, and overall GraphQL latency by 20%, with a double-digit decrease in CPU usage.


### [Engineering Principles at Ramp](https://yomu.fyi/post/engineering-principles-at-ramp.md)
- Company: [Ramp](https://yomu.fyi/company/ramp.md)
- Author: Calvin Lee
- Published: Sep 12, 2021
- Type: Explainer

Ramp documents seven engineering principles intended to codify how its teams build systems, make decisions, and collaborate as the company grows. The principles call for choosing technology according to each system’s context, moving aggressively on reversible decisions while treating durable interfaces cautiously, and aligning system models with customer concepts. They also favor opinionated product experiences, accepting duplication over a harmful abstraction while continuing to seek abstractions that reflect meaningful shared behavior. Rather than implementing requests literally, engineers investigate underlying needs, such as notifying a CFO instead of making approval a bottleneck, and break projects into small, independently prioritized chunks. These chunks can be owned by small teams called “pods,” while the principles remain adaptable rather than ironclad rules.


### [Using Rich Image and Text Data to Categorize Products at Scale](https://yomu.fyi/post/using-rich-image-and-text-data-to-categorize-products-at-scale.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Sep 8, 2021
- Type: Explainer

Shopify revisited product categorization as its platform expanded to millions of merchants selling billions of products across diverse industries. The model uses product titles, descriptions, types, vendors, collections, tags, and images, converting them into embeddings with Multilingual BERT and MobileNet-V2 before hierarchical classification across the seven-level Google Product Taxonomy. Each taxonomy level is treated as a separate multi-class task, with parent predictions fed into child predictions and weighted losses combined during training; confidence thresholds filter predictions at each level. The upgraded system increased leaf precision by 8% while almost doubling coverage, although thresholds require balancing accuracy, coverage, merchant experience, and performance in sensitive categories, and future work includes addressing data imbalance and adding merchant-level features.


### [A Kotlin Style .copy Function for Swift Structs](https://yomu.fyi/post/a-kotlin-style-copy-function-for-swift-structs.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Sep 2, 2021
- Type: Tutorial

Immutable state objects are used to keep rendering code from modifying shared values, supporting clearer value origins, fewer bugs, and more focused testing. Kotlin data classes provide a copy function that replaces selected fields while preserving the rest, allowing the new object to become the source of truth. Swift structs lack an equivalent convenience, and defining a complete initializer plus a parameter-based copy method becomes cumbersome, particularly with many or nested properties. Optional properties expose a further issue: nil must sometimes mean “set this field to nil,” not “use the existing value.” A functional builder closure solves that distinction by collecting overrides while retaining the original struct’s values for other fields, and a Sourcery stencil can generate the initializer and builder boilerplate.


### [Automating Multi-Armed Bandit testing during feature rollout](https://yomu.fyi/post/automating-multi-armed-bandit-testing-during-feature-rollout.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Weicheng Zhu
- Published: Sep 1, 2021
- Type: Problem & solution

Traditional feature rollouts and Multi-Armed Bandit testing operate as separate workflows that often depend on delayed offline analysis. To eliminate manual intervention, the Multi-Armed Bandit Optimiser automates testing concurrently during feature rollouts by responding to minute-level feedback metrics. The architecture connects Kafka Streams data processing, a metrics server with Spark jobs, and an adaptive rollout module updating online experimentation configurations. Candidate models are evaluated via Thompson Sampling on Beta distributions, with Monte Carlo simulations determining traffic allocation across user entities. In production for the GrabFood recommendation widget, the system optimizes the Effective Conversion Rate over a 30-minute window and includes fallback distribution logic.


### [Modelling Developer Infrastructure Teams](https://yomu.fyi/post/modelling-developer-infrastructure-teams.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Aug 13, 2021
- Type: Explainer

Developer Infrastructure at Shopify spans local and cloud development environments, CI systems, deploys, and technology-specific foundations for web and mobile applications. The post models these teams against a three-phase develop–validate–deploy workflow, distinguishing horizontally integrated teams that own one phase from vertically integrated teams whose systems span phases for a particular technology. It uses this model to identify dependencies, gaps, differing scopes, collaboration patterns, and three work avenues—concentration, expansion, and interfacing—with the mission of improving and scaling the cycle. The analysis informed organizational changes, including a Client Foundations team, closer collaboration, and plans for workflow-wide user research focused on the developer journey from writing a patch to production.


### [Epsilon3 Self-Host Supabase To Revolutionize Space Operations](https://yomu.fyi/post/epsilon3-self-host-supabase-to-revolutionize-space-operations.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Published: Jul 26, 2021
- Type: Problem & solution

Epsilon3 digitizes paper procedures and telemetry data for high-stakes space missions and complex operational testing. Operating in heavily regulated environments requires strict security, ITAR compliance, and flexible deployment options ranging from AWS GovCloud to on-premises installations. To meet these demands without sacrificing developer velocity, Epsilon3 deployed a self-hosted Supabase configuration running against an Amazon RDS Postgres instance on AWS. This architecture gives the team real-time data streaming capabilities over Postgres so operators can track live procedure changes as they happen. In addition to reducing DevOps complexity, self-hosting Supabase enables Epsilon3 to leverage Postgres Row Level Security while ensuring full compliance across diverse deployment environments.


### [A Guide to Running an Engineering Program](https://yomu.fyi/post/a-guide-to-running-an-engineering-program.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Jul 21, 2021
- Type: Explainer

Shopify describes a playbook for running large engineering programs as platform complexity grows across products, tooling, and architecture. The framework begins with a Program Plan covering the problem statement, objectives, guiding principles, definition of done, risks, mitigation, staffing, scope, and a path to completion. Execution uses six-week cycles that revisit unfinished goals, regressions, emerging risks, and the planned route to completion; the described program spans 200 people, nine sub-organizations, and 92 projects. Supporting rituals include weekly updates, status checks, risk and escalation triage, cycle reviews, retrospectives, RFCs, and performance testing with Lua scripts orchestrated by Genghis. The post concludes that engineering program management must adapt to each organization and that a combination of scheduled and ad hoc rituals helped Shopify pursue its goals.


### [Perspectives on React Native from Three Shopify Developers](https://yomu.fyi/post/perspectives-on-react-native-from-three-shopify-developers.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Jul 16, 2021
- Type: Explainer

Three Shopify developers—web developer AJ Robidas, Android developer Michelle Fernandez, and iOS developer Ash Furrow—describe how React Native changes mobile development for people from different backgrounds. They say its React-based model lets web developers build for iOS and Android, helps native developers create interfaces faster and reach a wider audience, and supports collaboration across web, Android, and iOS teams. The developers discuss surprises including React Native’s small set of building blocks, fast UI feedback, native bridge modules, existing SDK integration, and apps that can be difficult to distinguish from Swift or Objective-C software. They also identify challenges such as mobile-specific performance concerns across the RN Bridge, the Metro and Node ecosystem, and the large dependency stack. At Shopify, Visual Studio Code, command-line tooling, internal foundations, workshops, and cross-disciplinary teams support adoption.


### [App Modularisation at Scale](https://yomu.fyi/post/app-modularisation-at-scale.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Amar Jain
- Published: Jul 13, 2021
- Type: Problem & solution

Grab transitioned its monolithic mobile application into a modular architecture to resolve increasing code conflicts, slow releases, and difficult team collaboration. The team decomposed the single module by establishing base infrastructure modules, shared UI and utility libraries, discrete feature modules, and bridge kit modules for inter-module communication. Dependency injection using Dagger ties these components together in the main app module while preventing feature modules from directly depending on one another. The architecture spans over 1,000 modules across the app, with more than 200 modules in the Grab Financial Group payments domain where over 95% of modules build in under 15 seconds. This approach accelerated Gradle CI and local builds through parallel compilation and caching, though it increased Gradle sync times, IDE memory usage, and configuration maintenance overhead.


### [Shopify-Made Patterns in Our Rails Apps](https://yomu.fyi/post/shopify-made-patterns-in-our-rails-apps.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Jul 13, 2021
- Type: Explainer

Shopify describes patterns and infrastructure adaptations used to keep its large Rails monolith maintainable as its organization and platform scale. Its dev command-line tool standardizes project setup, dependencies, services, environment variables, integrations, and command aliases across hundreds of active projects. Shopify Core uses pods to partition shops across independent databases and extends the shard concept across provisioning, deployment, load balancers, caching, and servers, while splitting code into domains without separate databases. The post also details safeguards including short-term Active Record migrations consolidated periodically into structure.sql, automated tests, a pod-aware maintenance task system, and a translation platform that commits translated strings through Git. Finally, Monorail imposes versioned JSON event schemas between Kafka producers and consumers and supports review and automated scrubbing of PII fields.


### [High Availability by Offloading Work Into the Background](https://yomu.fyi/post/high-availability-by-offloading-work-into-the-background.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Jul 8, 2021
- Type: Explainer

Unpredictable traffic spikes, slow third-party payment requests, and resource-intensive image processing can overwhelm an application, making it slow or unavailable; the text presents background jobs as a way to preserve meaningful user interactions. High availability is defined not merely as server uptime, but as users being able to interact with the application when needed. The application server places a task message in a queue and continues serving requests, while workers process jobs asynchronously, with support for retries, parallelization, prioritization, and event- or time-based scheduling. Offloading improves response times and buffers spikes, but changes immediate guarantees: an upload may be accepted before processing succeeds, and errors may be handled later. For distributed systems, data-based messaging such as RabbitMQ can connect different services, while Kafka's append-only, replayable event log supports shared consumption and event sourcing.


### [Reshaping Chat Support for Our Users](https://yomu.fyi/post/reshaping-chat-support-for-our-users.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Elisa Monacchi
- Published: Jul 7, 2021
- Type: Problem & solution

Grab transitioned from voice hotlines and third-party tools to an in-house native chat support system integrated into their CRM. The team validated the platform through an MVP and user shadowing to address session disconnections, agent context switching, and routing bottlenecks. To optimize support operations at scale, they introduced dynamic queue limits based on Little's law, machine learning autocomplete suggestions for agents, and duration timers with visual nudges. These enhancements reduced chat waiting times by 30%, unresponsive users by 7%, and overall chat handling duration by 22%.


### [Understanding GraphQL for Beginners–Part Two](https://yomu.fyi/post/understanding-graphql-for-beginners-part-two.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Jul 2, 2021
- Type: Tutorial

The second part of this beginner tutorial uses a food-focused Ruby on Rails application to show how GraphQL fields expose database data. It reviews the graphql directory, where mutations hold data-changing classes, types define returned structures, and food\_app\_schema.rb defines available queries. It distinguishes root fields, which are GraphQL entry points that define response structure, from object fields, which represent attributes, then demonstrates generating Food and Nutrition types and adding an all\_food root field with a non-null list return type. Through GraphiQL at localhost:3000/graphiql, it runs selective queries, adds nutrition to food, and introduces an id argument for retrieving one record; requests pass through graphql\_controller’s execute method and return JSON responses.


### [Querying Strategies for GraphQL Clients](https://yomu.fyi/post/querying-strategies-for-graphql-clients.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Jun 15, 2021
- Type: Tutorial

GraphQL clients can face slower, larger queries and difficult roll-outs as product screens gain features and data. The Orders & Fulfillments teams examined these issues while targeting a sub-one-second mobile page load on a reliable network. Starting with a product-list query that loads 100 products, the post recommends pagination with page-size/index controls and a hasNextPage field, plus performance tripwires to track loading time. It describes @include and @skip directives, runtime query construction, and chained queries for feature flags or parameters, while warning that chained requests add latency. For growing screens, it separates unrelated filters, permissions, and banners into parallel queries, enabling independent scaling, partial rendering, and less redundant pagination work, while cautioning that server load and partial-data state require monitoring and possible refactoring.


### [Deleting the Undeletable: Managing PII at Shopify - Shopify](https://yomu.fyi/post/deleting-the-undeletable-managing-pii-at-shopify-shopify.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Jun 10, 2021
- Type: Problem & solution

Shopify describes how its Privacy and Data Science & Engineering teams redesigned analytical event handling to make personal-information deletion workable at Shopify’s scale. The effort addressed a lack of guaranteed event schemas by introducing a human-readable JSON schematization system covering field types, version evolution, ownership, data controllers and subjects, and privacy instructions; more than 4,500 active schemas now govern events. Before warehouse storage, identifying data is either obfuscated into generalized values, such as masked IP addresses with city or country information, or tokenized into consistent random values whose mappings reside in a secured vault and vary by controller and subject. Removing a vault mapping makes warehouse copies of the token non-detokenizable, allowing deletion without warehouse operations; the production platform processes about 20 billion events daily, while tens to hundreds of millions of mappings are deleted daily.


### [Shipit! Presents: How We Write React Native Apps](https://yomu.fyi/post/shipit-presents-how-we-write-react-native-apps.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: May 27, 2021
- Type: Announcement

A May 19, 2021 Shipit! event brought together Colin Gray, Haris Mahmood, Guil Varandas, and Michelle Fernandez, developers setting Shopify’s React Native standards, to discuss writing performant apps. The Q&A covers accessibility and design guidance through Restyle and Polaris documentation; their responsive layouts avoid fixed sizing where possible and use breakpoints for fonts and spacing. They describe React Native state as the usual source of truth while native modules manage internal state and send updates through EventEmitter, with SQLite in Point of Sale as an exception that still has one source of truth. Jest handles unit and integration tests, Buildkite manages CI builds, nightly builds are automatically distributed to testers, and failing feature-branch tests block merges to main.


### [How Shopify Built An In-Context Analytics Experience](https://yomu.fyi/post/how-shopify-built-an-in-context-analytics-experience.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: May 25, 2021
- Type: Explainer

Shopify’s Analytics section gave merchants access to business data, but research and quantitative analysis found that they often moved between it and admin pages while making decisions. The team therefore designed an in-context experience for the Orders page, selecting metrics through workflow analysis, segmenting merchants by factors such as order volume and fulfillment process, and examining each metric’s usefulness, variability, and time horizon. Data scientists prototyped visualizations with real, segmented data and created a specification sheet so Web, Android, and iOS used consistent queries, data sources, fields, and aggregations; the project also tested batch models before streaming them. The launched experience surfaces order and fulfillment metrics for daily, seven-day, and thirty-day periods, with comparison indicators, and links to reports covering orders, returns, products, and fulfillment, shipping, and delivery times.


### [Building a Hyper Self-Service, Distributed Tracing and Feedback System for Rule & Machine Learning (ML) Predictions](https://yomu.fyi/post/building-a-hyper-self-service-distributed-tracing-and-feedback-system.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Warren Zhou
- Published: May 24, 2021
- Type: Problem & solution

Grab's Trust, Identity, Safety, and Security team processes billions of daily rule and machine learning decisions for fraud detection, safety, and identity checks. Earlier logging approaches using plain text Kibana logs and the ActionTrace library lacked structured formats, dynamic entity customization, and fine-grained access controls. To resolve these limitations, the team built Archivist, a centralized tracing, statistics, and feedback system. Archivist ingests events through an SDK into Kafka streams, buffers and routes data into Elasticsearch indices and Amazon S3, and provides a role-based user portal. The platform handles 80 million daily logs across roughly 50 business scenarios, reducing scenario onboarding times from days to minutes.


### [Three Ways We Share Context at Shopify Engineering](https://yomu.fyi/post/three-ways-we-share-context-at-shopify-engineering.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: May 13, 2021
- Type: Explainer

Shopify Engineering treats context—the why, what, and how behind development philosophies, approaches, and choices—as essential to company, team, and individual growth. It describes the Development Handbook, which evolved from a 2015 Google Doc into a searchable internal website in 2018, with hundreds of developer-contributed topics maintained through GitHub pull requests, reviews, and Markdown. Dev Talks began as a weekly open stage and became an on-demand recording program after the 2020 shift to Digital by Design, with most presentations lasting 15 to 20 minutes. The Technical Leadership Team’s Shift podcast provides monthly, in-depth discussions of technical decisions, distributed in audio and video with transcripts. The programs continue to evolve while facing content-maintenance and participation challenges.


[Newer posts](https://yomu.fyi/topic/architecture/page/36.md) · [Older posts](https://yomu.fyi/topic/architecture/page/38.md)