---
title: "Architecture"
description: "834 posts about Architecture, summarised, each linking to the original."
---

# Architecture
> 834 posts about Architecture, summarised, each linking to the original.

## Articles

### [Improving the Developer Experience with the Ruby LSP](https://yomu.fyi/post/improving-the-developer-experience-with-the-ruby-lsp.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Feb 24, 2023
- Type: Explainer

The Ruby LSP is a language server intended to improve Ruby’s developer experience by providing editor features through the Language Server Protocol, without typechecking code or requiring type annotations. It targets performance and stability in Shopify’s Core monolith, where it must analyze thousands of files that may themselves contain thousands of lines, and is opinionated toward Ruby on Rails. The article explains LSP’s client-server model, in which editors exchange JSON requests and responses over STDIN and STDOUT, then focuses on nonpositional features such as Folding Range, including document synchronization and parsing. It also outlines future work on parallel request processing, codebase indexing, caching, and a plugin system for framework-specific capabilities such as Rails navigation and model information.


### [The 25 Percent Rule for Tackling Technical Debt](https://yomu.fyi/post/the-25-percent-rule-for-tackling-technical-debt.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Feb 16, 2023
- Type: Explainer

Technical debt is framed as four categories that need different responses: yearly debt may involve rewrites or gradual replacement, monthly debt requires team-sized planning, weekly debt can become a sprint card, and daily debt is code tidying during ordinary work. The article cautions that some apparent yearly debt reflects growth or changing markets rather than a past mistake, while smaller debt can be deferred only if it is recorded and scheduled. The proposed 25-percent allocation gives 10% to daily improvements, 10% to weekly work, and 5% to planning monthly and yearly debt, with the latter focused on prioritization rather than implementation. Its conclusion is that sustainable remediation depends on team culture: engineers need permission and encouragement to refactor, teams need space to schedule broader work, and leaders need to weigh costs and tradeoffs.


### [ShopifyQL Notebooks: Simplifying Querying with Commerce Data Models](https://yomu.fyi/post/shopifyql-notebooks-simplifying-querying-with-commerce-data-models.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Feb 14, 2023
- Type: Tutorial

ShopifyQL Notebooks gives merchants a self-serve way to investigate changing business metrics, understand their causes, and share findings as data stories. It combines the ShopifyQL query language with commerce-specific data models designed for users without SQL expertise, covering domains such as products, orders, and customers. The modeling process starts with business questions, translates them into columns and an appropriate grain, validates the design with mock data and queries, and evaluates upstream data, freshness, and performance. To support large merchants, the implementation uses incremental models and roll-ups, which can reduce build and query times while adding maintenance, pipeline, and monitoring complexity.


### [The Complex Data Models Behind Shopify's Tax Insights Feature](https://yomu.fyi/post/the-complex-data-models-behind-shopify-s-tax-insights-feature.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Feb 8, 2023
- Type: Explainer

Shopify Tax’s Tax Insights feature provides merchants with state-level information about sales-tax liability and approaching liability, amid U.S. rules that vary by jurisdiction, product, threshold, and evaluation period. The implementation modified existing data models, added four new ones, and represented changing tax rules with lookup data containing valid\_from and valid\_to dates, including thresholds, date ranges, inclusions, exclusions, and inclusive or exclusive comparisons. Teams prototyped models in BigQuery SQL, implemented them in PySpark and Spark jobs, and combined merchant monthly taxable-sales aggregates with the rule model to evaluate economic nexus; the architecture moved source data through GCS and a warehouse into a final model. Each run publishes merchant-keyed insights to Google Cloud Bigtable, where the core application retrieves them for the frontend, while warehouse data supports analytics and other processes.


### [Carving The Scheduler Out Of Our Orchestrator](https://yomu.fyi/post/carving-the-scheduler-out-of-our-orchestrator.md)
- Company: [Fly.io](https://yomu.fyi/company/fly-io.md)
- Published: Feb 1, 2023
- Type: Explainer

Fly.io describes why it is carving scheduling out of its existing Nomad-based orchestrator and replacing it with a simpler system centered on flyd and AppsV2. It first reduces orchestration to a client-server supervisor, then explains centralized scheduling: workers register resources, while a server filters candidates by constraints and ranks them. The proposed design avoids jobs sitting pending or being placed in nearby but incorrect regions; requesting VMs in MAD yields MAD capacity or nothing. Policy-heavy behavior such as deployment spares, crash recovery across workers, and regional decisions moves into flyctl and other platform components rather than the scheduler. Fly Machines already exposes APIs to reserve, start, and stop individual VMs, with Nomad still used in January 2023 and flyd expected for deployments in December 2023.


### [From Ruby to Node: Overhauling Shopify’s CLI for a Better Developer Experience](https://yomu.fyi/post/from-ruby-to-node-overhauling-shopify-s-cli-for-a-better-developer-exp.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Jan 10, 2023
- Type: Problem & solution

Shopify rewrote its unified CLI from Ruby to TypeScript running on Node after growing dependencies on Node tooling, Hydrogen’s preference for npm-managed dependencies, and a broader fit with web technologies. The rewrite aims to reduce system requirements, improve terminal consistency, and make contribution easier, while using a modular architecture of domain NPM packages built on @shopify/cli-kit. The team established shared code, UI, convention, and principle foundations, addressed cross-OS support, used TypeScript for stronger contracts, and applied Cucumber for end-to-end testing. The Node CLI unified and streamlined app development and added extensibility such as functions, but Theme commands still use Ruby and removing that dependency remains future work.


### [Graph service platform](https://yomu.fyi/post/graph-service-platform.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Wenxiang Lu
- Published: Jan 5, 2023
- Type: Problem & solution

Grab's GrabDefence team required a dedicated graph infrastructure to proactively identify mobile fraud patterns, such as multiple accounts operating on shared physical devices and suspicious financial loops. To address this, the team built a four-layer Platform as a Service that encapsulates graph database operations behind uniform RESTful APIs for OLTP search and OLAP analysis. The architecture utilizes Amazon S3 for raw data files, Amazon Neptune for graph storage, DynamoDB for schema and metadata configurations, and Kafka for streaming ingestion. When users trigger data loading tasks, the service validates entity attributes against schemas stored in DynamoDB before importing records into Neptune. This infrastructure allows investigators to traverse adjacent account IDs and visualize complex entity relationships without managing underlying database runtimes.


### [React Native Skia: A Year in Review and a Look Ahead](https://yomu.fyi/post/react-native-skia-a-year-in-review-and-a-look-ahead.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Dec 29, 2022
- Type: Explainer

React Native Skia’s first year produced an initial alpha and 89 additional releases, evolving a React Renderer and Skia integration for declarative 2D graphics across iOS, Android, the Web, and Node.js testing. The project changed direction after identifying two bottlenecks: dependence on the JavaScript thread and the overhead of thousands of JSI allocations in large drawings. Skia DOM addresses them with a declarative drawing state that recomputes only necessary changes and executes drawings on the UI thread, with C++ implementations on iOS and Android and a JavaScript implementation on React Native Web. CanvasKit compatibility, Expo integration, Metal on iOS, and OpenGL on Android and the Web support the cross-platform design, while extension points enable modules such as Skottie and camera-frame effects. The library’s tests compare rendered examples across platforms, and planned work includes JavaScript-worklet and Reanimated animation integration plus advanced text layouts through Skia’s paragraph module.


### [Migrating our Largest Mobile App to React Native](https://yomu.fyi/post/migrating-our-largest-mobile-app-to-react-native.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Dec 27, 2022
- Type: Problem & solution

Shopify’s largest mobile app, Shopify Mobile, had 300 screens per platform and more than six years of development, making a full rewrite impractical because it could take at least three years and halt feature work. The team first adopted React Native incrementally, but migration was projected to take four to five years while requiring three design-system implementations, three architectures, and costly native interoperability. It then switched to “Iterative Porting”: engineers were trained through the five-milestone RN Accelerator, surfaces were inventoried and prioritized with a RICE-based score, and eligible features were migrated behind flags, tested, monitored for two weeks, then completed by removing the native version. The effort also introduced Native Routes, modal deeplink handling on iOS, and component-level Safe Area fixes; with root screens ported and infrastructure in place, the team reports that ports are accelerating, although substantial work remains.


### [Optimizing Ruby’s Memory Layout: Variable Width Allocation](https://yomu.fyi/post/optimizing-ruby-s-memory-layout-variable-width-allocation.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Dec 25, 2022
- Type: Explainer

CRuby’s original garbage collector stores most Ruby objects in fixed 40-byte RVALUE slots, leaving differently sized data partly unused or forcing additional allocations through malloc. Shopify’s Variable Width Allocation project adds dynamic-sized allocation APIs and five fixed-size pools—40, 80, 160, 320, and 640 bytes—while increasing heap pages from 16 KB to 64 KB. The design targets cache locality, malloc overhead, memory utilization, and future garbage-collector experimentation; a new GC.stat\_heap API reports size-pool statistics, and an escape hatch can disable the feature at build time on Ruby 3.2. Benchmarks on development Ruby 3.2.0 showed gains in most workloads, including 9.8% for psych-load, 5.3% for hexapdf, and 2.1% for railsbench, while mail was 1.7% slower.


### [pg\_graphql v1.0](https://yomu.fyi/post/pg-graphql-v1-0.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: oli\_rice
- Published: Dec 16, 2022
- Type: Announcement

pg\_graphql 1.0 marks the general availability of a PostgreSQL extension that lets databases be queried with GraphQL and formalizes a SemVer-based guarantee against backward-incompatible changes. It was designed for Supabase Free Plan servers with 1 GB of memory, where alternatives such as Hasura and Graphile consumed more memory than the platform could sacrifice. The implementation began with a C parser and SQL transpiler exposed through PostgREST RPC, then moved from SQL to Rust after nested SQL functions, copied state, and JSONB AST processing slowed schema introspection; v0.5.0 completed the rewrite. On Free Plan hardware, version 1.0 adds less than 300 microseconds per request, reaches about 1,060 SQL queries per second per connection, and the full Auth and HTTP stack handles about 645 requests per second, while larger instances exceed 10,000 requests per second.


### [Point in Time Recovery is now available for Pro projects](https://yomu.fyi/post/point-in-time-recovery-is-now-available-for-pro-projects.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: angelico\_de\_los\_reyes
- Published: Dec 16, 2022
- Type: Announcement

Point-in-Time Recovery (PITR) is now available to eligible Pro Plan projects, expanding access beyond Enterprise customers and adding a Dashboard UI for enabling it. Projects created from August 2022 onward can enable PITR except in Northeast Asia / Seoul, and they must use the Small compute add-on; older projects are not yet generally eligible. The system combines daily physical database snapshots with archived Write Ahead Log (WAL) records, using the open-source WAL-G tool; snapshots are uploaded daily and WAL files are archived as transactions occur. WAL files are backed up at two-minute intervals, potentially faster when a file threshold is reached, reducing the recovery point from the up-to-24-hour exposure of daily backups. PITR backups retain up to seven days by default, with self-serve expansion to 28 days, and restoration can target a specified point in time.


### [Supabase Wrappers, a Postgres FDW framework written in Rust](https://yomu.fyi/post/supabase-wrappers-a-postgres-fdw-framework-written-in-rust.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: paul\_copplestone,bo\_lu,oli\_rice
- Published: Dec 15, 2022
- Type: Announcement

Supabase Wrappers is an Alpha framework for building PostgreSQL Foreign Data Wrappers that connect databases with external systems, including third-party APIs, through SQL. It initially supports Firebase and Stripe, while wrappers for ClickHouse, BigQuery, and Airtable are under development. The framework extends pgx, a Rust Postgres extension framework, with async support for RESTful API-based wrappers and limited push-down support currently provided as a proof of concept. The post presents Wrappers as a basis for QETL, allowing teams to query, extract, transform, and load remote data through Postgres, either on demand or into local tables, while joining it with operational data. It is released outside official platform support, with potential breaking changes.


### [Supabase Storage v2: Image resizing and Smart CDN](https://yomu.fyi/post/supabase-storage-v2-image-resizing-and-smart-cdn.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: fabrizio,inian
- Published: Dec 13, 2022
- Type: Announcement

Supabase Storage introduces image resizing, webhooks, and a Smart CDN, positioning them as components of an on-demand image delivery system. Resizing uses imgproxy and supports width, height, and contain, cover, or fill modes through JavaScript clients or direct Storage API requests; images are not enlarged beyond their original dimensions, and aspect ratio is preserved by default. The Smart CDN stores resized results using cache keys based on an Etag, tenant ID, bucket, and image name, while webhook-driven metadata updates propagate invalidations globally within about 60 seconds. Image Resizing is beta for Pro and Enterprise customers, with 100 transformed images per project at no additional cost and $5 per additional 1,000; self-hosting is free.


### [New Supabase Docs, built with Next.js](https://yomu.fyi/post/new-supabase-docs-built-with-next-js.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: jonny
- Published: Dec 12, 2022
- Type: Announcement

Supabase released a new version of its documentation, rebuilding the site with Next.js after outgrowing Docusaurus. The earlier setup became harder to maintain as client-library versioning expanded, contributions grew more difficult, and the docs remained separate from Supabase’s Next.js and Turborepo monorepo. To reduce migration risk, the team first created a one-week Next.js MVP that copied Docusaurus, then spent a month developing the new release with two full-time developers. It introduces a new design, a single sidebar for broader discoverability, and reference pages that group all functions in a library while improving TypeDoc-based generation. Future work described includes unified authentication, project-specific documentation, runnable Postgres WASM examples, interactive tutorials, improved Postgres full-text search, and balancing sparse and overly verbose content.


### [pg\_crdt - an experimental CRDT extension for Postgres](https://yomu.fyi/post/pg-crdt-an-experimental-crdt-extension-for-postgres.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: paul\_copplestone
- Published: Dec 10, 2022
- Type: Announcement

Supabase announces pg\_crdt, an experimental Postgres extension that adds CRDT support as a data type for collaborative and offline-oriented applications. It frames CRDTs as structures whose updates can be merged in any order to reach the same final state, avoiding the last-write-wins behavior of ordinary arrays. A content column can use crdt.ydoc, backed by Yjs, while the extension also supports Automerge through Rust libraries wrapped with pgx. The post contrasts storing CRDTs directly in Postgres with making Realtime an authoritative server, favoring the database-centered approach for simpler architecture and broader tooling, while noting that the decision is unsettled and pg\_crdt is not released on the Supabase platform. Known limitations include full-document WAL broadcasts, increased WAL and dead tuples for frequently updated documents, and serialization overhead for large CRDT values.


### [Who We Hire at Supabase](https://yomu.fyi/post/who-we-hire-at-supabase.md)
- Company: [Supabase](https://yomu.fyi/company/supabase.md)
- Author: ant\_wilson
- Published: Dec 9, 2022
- Type: Explainer

Supabase describes the hiring traits and operating model it believes support its growth, arguing that people matter more than methodology. It looks for extreme asynchronous effectiveness, egoless execution, continuous incremental improvement, a bias toward immediate action, ambitious problem-solving, and a founder mindset; seniority does not exempt anyone from independent-contributor work, self-management, or maintenance. The 60-person team spans 25 countries, uses meetings as a last resort, and gives autonomous product teams access to shared metrics, user feedback, and equity through an ESOP. Teams are organised around products such as Postgres, Auth, Storage, Realtime, and Functions, while cross-cutting groups consult rather than merely receive delegated tasks. Three-month product cycles set ambitious goals, and Kaizen Weeks focused on QA, documentation, and performance help address technical debt and incorporate improvements into ongoing workflows.


### [Zero trust with Kafka](https://yomu.fyi/post/zero-trust-with-kafka.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Fabrice Harbulot
- Published: Dec 7, 2022
- Type: Problem & solution

Grab's real-time data platform team transitioned their large-scale Kafka infrastructure from basic network access controls to a zero-trust architecture. The platform implements mutual Transport Layer Security (mTLS) for offline peer authentication and encryption, driven by HashiCorp Vault's PKI engine and Strimzi on Kubernetes. Policy-Based Access Control is enforced using dedicated Open Policy Agent deployments per cluster, backed by GitOps workflows where topic owners approve JSON authorization rules. To simplify client integration, the team enhanced their Go SDK to handle ephemeral in-memory certificates, automatic renewals, and configurable retries. While the security posture improved, the Java encryption and decryption overhead caused a drop in streaming throughput.


### [Planning in Bets: Risk Mitigation at Scale](https://yomu.fyi/post/planning-in-bets-risk-mitigation-at-scale.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Dec 6, 2022
- Type: Explainer

Preparing for BFCM and other high-traffic events, Shopify describes risk mitigation for a globally distributed, complex, interconnected infrastructure platform that peaked at 75.98M requests per minute, or 1.27M per second. At this scale, identifying and mitigating every possible failure is impossible, so the process starts with “what could go wrong” exercises, then uses voting, expert review of likelihood and severity, and assigned ownership to prioritize action. Decision makers are empowered to weigh risks and rewards, while summarized findings, including key risks and single points of failure, are shared to maintain alignment and awareness. The post frames engineering choices as probabilistic bets: decision quality should be judged separately from outcomes, and uncertainty cannot be eliminated, only addressed through thoughtful prioritization and communication.


### [Graph for fraud detection](https://yomu.fyi/post/graph-for-fraud-detection.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Min Chen
- Published: Nov 24, 2022
- Type: Problem & solution

Rapid expansion across multiple business verticals introduced diverse, evolving fraud patterns at Grab, making traditional rule engines and decision trees inadequate due to their heavy reliance on extensive feature engineering and labeled data. To address cold starts and detect unknown fraud, the engineering team implemented a semi-supervised Relational Graph Convolutional Network (RGCN) across millions of connected ecosystem entities. The model propagates information along neighborhood nodes through graph convolutional layers to produce node embeddings and output fraud probabilities. Trained on graphs where only a small percentage of nodes are labeled, the RGCN achieved an AUROC close to 1 and distinctly separated fraudulent embeddings from genuine ones. The approach also offers high explainability by surfacing dense clusters of shared physical devices and addresses ongoing challenges in real-time prediction and noisy connections.


[Newer posts](https://yomu.fyi/topic/architecture/page/31.md) · [Older posts](https://yomu.fyi/topic/architecture/page/33.md)