Loading…
Topic
824 posts about Architecture. Every summary links to the original.
See the idea
Argo CD is a GitOps controller that treats Git as the desired state and continuously reconciles it against what actually runs in each cluster. It maps the parts of that loop and why pull-based agents change its scale.
AI data assimilation fuses a model's prior state with fresh, uneven observations into one consistent state. See the four-layer architecture and where each layer can break.
See how Bazel’s dependency graph can separate build scope from IDE scope, helping engineers move through large monorepos without syncing every module.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Supabase updated its Realtime server to enforce PostgreSQL Row Level Security (RLS) policies when broadcasting database changes over websockets. Previously, Realtime operated as an opt-in beta feature that sent all replication changes to every client regardless of user authorization. To enforce RLS per subscriber without heavy performance overhead, Supabase introduced WALRUS, a security engine colocated inside PostgreSQL. For each replication change, WALRUS looks up active subscribers, assumes their identities, and evaluates row visibility using prepared statements queried by primary key. This in-database evaluation avoids external network round trips and single-query planning overhead while returning an authorized subscriber list to Realtime.
Shopify describes a UI extension architecture designed to give third-party code consistent, accessible, cross-platform experiences without allowing it to compromise the host application. Its remote-rendering approach separates UI definition from rendering, with extensions communicating with hosts through MessageChannel-based message passing; hosts are implemented in JavaScript, Kotlin, and Swift. The open-source remote-ui library supplies RPC for passing callbacks across postMessage, automatic memory management, a DOM-like RemoteRoot that serializes updates as JSON, and receivers that map remote components to DOM or React components. A sandbox isolates untrusted extension code and controls exposed APIs across web workers, Android web views, and iOS JsCore. The system powers product-subscription and post-purchase extensions, while a DOM-based version began releasing in July 2025.
The Shop team built an iOS App Clip to reduce the drop-off between a buyer’s checkout thank-you page and opening the full Shop app, where App Store download and onboarding added friction. App Clips are invoked through actions such as QR scans, links, or Smart App Banners and must stay below 10MB, while the team wanted React Native support for order tracking and related functionality. A Hello World build initially measured 28MB; explicitly selecting native dependencies in the Podfile reduced the baseline to 4.3MB, and a scrappy feature build reached 6.5MB. The team used early testing, invocation URL handling through React Native Linking, and shared data containers for app upgrades, ultimately releasing an App Clip at 9.1MB; extensive code sharing proved too complex and some code was copied.
Continuous ride-hailing assignment relies on solving the minimum weight bipartite matching problem between passengers and driver-partners. While traditional implementations assume a precalculated cost matrix, computing shortest-path travel times across large road networks dominates total execution time. Researchers introduced an Incremental Kuhn-Munkres algorithm that leverages the spatial locality of optimal matches to compute edge costs on demand. The approach integrates priority queues and lower-bounding techniques with refinement rules to avoid evaluating distant pairs while guaranteeing the same optimal assignment. Evaluated on Singapore road network data and real Grab production workloads, the incremental techniques reduced exact cost calculations and decreased assignment running times by over an order of magnitude.
Ryu is a component library created at a small startup to separate design concerns from feature development and business logic while product and conventions continue changing. It mirrors Figma's design system through shared colors, sizes, fonts, line heights, media breakpoints, z-indices, and components such as inputs, buttons, tooltips, menus, tables, and drawers. The article argues for a hard boundary between the library and main codebase, using a separate repository and an npm package with documented interfaces, hand-crafted changelogs, versioned releases, and extensive TypeScript. It emphasizes resisting narrowly tailored settings, postponing unresolved use cases, and prioritizing adoption over completeness, including preventing new code from using legacy components without good reason. Building on mature existing components while simplifying their interfaces, the library aims to improve maintainability, consistency, accessibility, theming, contribution, and coordination across applications.
Hydrogen is a React-based framework for custom Shopify storefronts, designed to balance fast loading, personalized and dynamic commerce, and developer experience. The central trade-off is that static generation and edge delivery can speed loading but defer personalization, while conventional server-side rendering can slow time-to-first-byte as data is fetched. Hydrogen addresses this with React 18 alpha streaming SSR powered by Suspense, which streams the page shell before data resolves and progressively hydrates components without extra client round trips. React Server Components keep server-only code out of bundles, support server-side data access, and enable component-level updates that preserve client state. Built-in fetching and configurable caching, including stale-while-revalidate, work with Oxygen’s colocated runtime, while developers can mix server rendering, client fetching, and edge delivery at page or component level.
Recent findings from the 2021 State of AI Report and Kaggle State of Machine Learning and Data Science Survey indicate that machine learning is expanding into critical infrastructure while Transformers become general-purpose architectures across text, vision, and audio. In response, organizations face questions about scaling infrastructure, team composition, and engineering maturity. Rather than treating machine learning as isolated sandbox experiments or hiring solely data scientists, teams benefit from adopting established software engineering and DevOps principles like versioning, testing, automation, and continuous deployment. Furthermore, the rise of pre-trained Transformer architectures enables practitioners to fine-tune existing off-the-shelf models rather than training from scratch, reducing compute costs and training duration. Tools from platforms such as Hugging Face streamline model deployment, latency optimization, and infrastructure abstraction.
YJIT is a new just-in-time compiler being built inside CRuby to improve Ruby performance on modern hardware, where single-core speed gains are limited and energy efficiency matters. Unlike MJIT’s weaker results on large real-world applications, YJIT uses a data-driven focus on performance hotspots in Rails and Shopify Core and is based on Basic Block Versioning (BBV). After about one year, it reports speedups over the CRuby interpreter of 20% on railsbench, 39% on liquid template rendering, and 37% on activerecord, with near-peak performance after one benchmark iteration. YJIT passes CRuby’s roughly 30,000-test suite, Shopify Core and GitHub backend CI, while running on a small percentage of Shopify production servers; its team plans further CRuby changes and broader JIT coverage before and after its intended Ruby 3.1 integration.
Shopify’s React Native teams needed to reuse shared functionality across Shop, Inbox, Point of Sale, Shopify Mobile, and Local Delivery without requiring each team to build and maintain separate package infrastructure. The React Native Foundations team chose a monorepo containing npm packages, a fixture app, shared root configuration, generated CI, and an internal documentation site, with code generation for package bootstrapping. Packages use TypeScript project references, Yarn Workspaces, Lerna publishing, and custom Babel, Metro, and Podfile configuration to support cross-package references, hot reload, and local native dependencies. Dependency graphs and file-change triggers limit CI work to affected packages and dependents, while independent versioning and release scripts manage publication; since June 2020, 17 packages have been developed, including 10 contributed by product teams, although release automation and coordination across multiple monorepos remain future improvements.
Shopify’s MySQL shards can become unevenly utilized as merchants grow after being assigned to arbitrary shards, increasing over-utilization risk while leaving other capacity underused. The post describes a strategy for selecting shop placements using historical utilization and traffic data, simulations, and potentially other metrics such as shop size, GMV, and move time. It then explains online shop migration with Ghostferry, which batch-copies records and tails MySQL’s binary log before a brief cutover updates the routing table and directs new writes to the target pod. Verification checks data completeness, integrity, and stale traffic before old data is pruned; the migration system’s central algorithm is also represented in a TLA+ formal specification. The stated outcome is a more balanced infrastructure that can isolate high-traffic merchants across pods with virtually zero consumer-facing downtime.
Shop’s home feed, the app’s most used feature, aggregated orders from millions of Shopify and non-Shopify merchants plus tracking data from dozens of carriers, accounted for 30% of database load and affected application performance. Before each database update, the custom write-through cache marks a user’s cache invalid, applies the write, then refreshes and revalidates it. To handle concurrent updates, it uses a separate pending-writes key and a short-expiry key to distinguish active writes from interrupted processes, while Active Record Concerns preserve the existing ORM interface. After a staged validation rollout, the cache reduced database load by 15%, overall app latency by about 20%, and overall GraphQL latency by 20%, with a double-digit decrease in CPU usage.
Ramp documents seven engineering principles intended to codify how its teams build systems, make decisions, and collaborate as the company grows. The principles call for choosing technology according to each system’s context, moving aggressively on reversible decisions while treating durable interfaces cautiously, and aligning system models with customer concepts. They also favor opinionated product experiences, accepting duplication over a harmful abstraction while continuing to seek abstractions that reflect meaningful shared behavior. Rather than implementing requests literally, engineers investigate underlying needs, such as notifying a CFO instead of making approval a bottleneck, and break projects into small, independently prioritized chunks. These chunks can be owned by small teams called “pods,” while the principles remain adaptable rather than ironclad rules.
Shopify revisited product categorization as its platform expanded to millions of merchants selling billions of products across diverse industries. The model uses product titles, descriptions, types, vendors, collections, tags, and images, converting them into embeddings with Multilingual BERT and MobileNet-V2 before hierarchical classification across the seven-level Google Product Taxonomy. Each taxonomy level is treated as a separate multi-class task, with parent predictions fed into child predictions and weighted losses combined during training; confidence thresholds filter predictions at each level. The upgraded system increased leaf precision by 8% while almost doubling coverage, although thresholds require balancing accuracy, coverage, merchant experience, and performance in sensitive categories, and future work includes addressing data imbalance and adding merchant-level features.
Immutable state objects are used to keep rendering code from modifying shared values, supporting clearer value origins, fewer bugs, and more focused testing. Kotlin data classes provide a copy function that replaces selected fields while preserving the rest, allowing the new object to become the source of truth. Swift structs lack an equivalent convenience, and defining a complete initializer plus a parameter-based copy method becomes cumbersome, particularly with many or nested properties. Optional properties expose a further issue: nil must sometimes mean “set this field to nil,” not “use the existing value.” A functional builder closure solves that distinction by collecting overrides while retaining the original struct’s values for other fields, and a Sourcery stencil can generate the initializer and builder boilerplate.
Traditional feature rollouts and Multi-Armed Bandit testing operate as separate workflows that often depend on delayed offline analysis. To eliminate manual intervention, the Multi-Armed Bandit Optimiser automates testing concurrently during feature rollouts by responding to minute-level feedback metrics. The architecture connects Kafka Streams data processing, a metrics server with Spark jobs, and an adaptive rollout module updating online experimentation configurations. Candidate models are evaluated via Thompson Sampling on Beta distributions, with Monte Carlo simulations determining traffic allocation across user entities. In production for the GrabFood recommendation widget, the system optimizes the Effective Conversion Rate over a 30-minute window and includes fallback distribution logic.
Developer Infrastructure at Shopify spans local and cloud development environments, CI systems, deploys, and technology-specific foundations for web and mobile applications. The post models these teams against a three-phase develop–validate–deploy workflow, distinguishing horizontally integrated teams that own one phase from vertically integrated teams whose systems span phases for a particular technology. It uses this model to identify dependencies, gaps, differing scopes, collaboration patterns, and three work avenues—concentration, expansion, and interfacing—with the mission of improving and scaling the cycle. The analysis informed organizational changes, including a Client Foundations team, closer collaboration, and plans for workflow-wide user research focused on the developer journey from writing a patch to production.
Epsilon3 digitizes paper procedures and telemetry data for high-stakes space missions and complex operational testing. Operating in heavily regulated environments requires strict security, ITAR compliance, and flexible deployment options ranging from AWS GovCloud to on-premises installations. To meet these demands without sacrificing developer velocity, Epsilon3 deployed a self-hosted Supabase configuration running against an Amazon RDS Postgres instance on AWS. This architecture gives the team real-time data streaming capabilities over Postgres so operators can track live procedure changes as they happen. In addition to reducing DevOps complexity, self-hosting Supabase enables Epsilon3 to leverage Postgres Row Level Security while ensuring full compliance across diverse deployment environments.
Shopify describes a playbook for running large engineering programs as platform complexity grows across products, tooling, and architecture. The framework begins with a Program Plan covering the problem statement, objectives, guiding principles, definition of done, risks, mitigation, staffing, scope, and a path to completion. Execution uses six-week cycles that revisit unfinished goals, regressions, emerging risks, and the planned route to completion; the described program spans 200 people, nine sub-organizations, and 92 projects. Supporting rituals include weekly updates, status checks, risk and escalation triage, cycle reviews, retrospectives, RFCs, and performance testing with Lua scripts orchestrated by Genghis. The post concludes that engineering program management must adapt to each organization and that a combination of scheduled and ad hoc rituals helped Shopify pursue its goals.
Three Shopify developers—web developer AJ Robidas, Android developer Michelle Fernandez, and iOS developer Ash Furrow—describe how React Native changes mobile development for people from different backgrounds. They say its React-based model lets web developers build for iOS and Android, helps native developers create interfaces faster and reach a wider audience, and supports collaboration across web, Android, and iOS teams. The developers discuss surprises including React Native’s small set of building blocks, fast UI feedback, native bridge modules, existing SDK integration, and apps that can be difficult to distinguish from Swift or Objective-C software. They also identify challenges such as mobile-specific performance concerns across the RN Bridge, the Metro and Node ecosystem, and the large dependency stack. At Shopify, Visual Studio Code, command-line tooling, internal foundations, workshops, and cross-disciplinary teams support adoption.
Grab transitioned its monolithic mobile application into a modular architecture to resolve increasing code conflicts, slow releases, and difficult team collaboration. The team decomposed the single module by establishing base infrastructure modules, shared UI and utility libraries, discrete feature modules, and bridge kit modules for inter-module communication. Dependency injection using Dagger ties these components together in the main app module while preventing feature modules from directly depending on one another. The architecture spans over 1,000 modules across the app, with more than 200 modules in the Grab Financial Group payments domain where over 95% of modules build in under 15 seconds. This approach accelerated Gradle CI and local builds through parallel compilation and caching, though it increased Gradle sync times, IDE memory usage, and configuration maintenance overhead.
The best engineering reads, once a week.
