API Versioning as a Translation Layer
API versioning as an architecture: requests enter through a version header, pass through a chain of reversible version changes, and reach controllers that only ever see the latest model.
Loading…
Topic
Architecture
830 posts about Architecture. Every summary links to the original.
- 830 posts
- 50 companies
- 31 explainers
See the idea
Architecture explainers
BIAN as a Contract Layer: Modernizing Core Banking One Domain at a Time
BIAN gives banks a shared vocabulary and service-domain contracts that sit between business concepts and the systems implementing them. See how domains, contracts, and governed access stack into one architecture.
Argo CD: The Reconciliation Loop Between Git and Your Cluster
Argo CD is a GitOps controller that treats Git as the desired state and continuously reconciles it against what actually runs in each cluster. It maps the parts of that loop and why pull-based agents change its scale.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
Caching Without Marshal Part 2: The Path to MessagePack
Rails caching relied heavily on Marshal, but cached Ruby-specific objects could break after code changes when old payloads were loaded by code without the former classes. MessagePack provided a generic binary format whose core types exclude Ruby-specific Object and instance-variable encodings, while extension types allowed production-specific serializers such as the compact Date representation. During a roughly six-month migration, the team ran both formats, prefixed MessagePack payloads with a version byte, and used encoding failures to identify unsupported cache values. To scale beyond 128 extension types, they added an Object catchall based on as_pack and from_pack, plus Struct and T::Struct modules whose attribute digests detect refactors and turn stale data into cache misses. After logs showed that Marshal was no longer used, the core monolith moved exclusively to MessagePack, with the Paquito gem extracting much of the migration work.
Caching Without Marshal Part 1: Marshal from the Inside Out
Rails caching commonly relies on Ruby’s Marshal serialization, which can encode complex objects but also embeds their class names in cache entries. Shopify describes an incident in which a beta-flag refactor changed classes while old and new code overlapped during deployment; old code then failed on cached instances containing unfamiliar class names and methods, despite passing CI. The article examines Marshal through Ruby’s marshal.c implementation, covering atomic and composite types, instance variables, object references, circularity via TYPE_LINK, and core-type subclasses via TYPE_UCLASS. It presents MessagePack as a more compact, stricter, and controllable alternative intended to make caching safer by default, while deferring the cache migration and implementation details to the series’ next part.
Authentication in Ionic Angular with Supabase
The tutorial builds a simple realtime messaging app with Ionic Angular and Supabase, covering account authentication and database protection. It defines Postgres tables for users, groups, and messages, enables Row Level Security with policies for public reads and authenticated operations, and adds a trigger to copy newly registered users into a public users table. Angular services, route guards, and Supabase client logic handle email/password registration and login, session state, protected navigation, and magic links. For native apps, Capacitor URL events extract access and refresh tokens from a custom-scheme callback before establishing the session and navigating to the groups page. The tutorial concludes that Supabase can support Ionic Angular authentication with magic links, while phone authentication through Twilio is another possible option.
The Management Poles of Developer Infrastructure Teams
Developer infrastructure managers must balance three poles: management support, system and domain expertise, and organization-wide road maps, with the tension intensified by teams that often lack dedicated product managers and support critical systems for hundreds or thousands of users with six to eight developers. Domain-focused teams build deep knowledge needed for maintenance, migrations, incident investigation, on-call coverage, and developer-experience improvements, but their local road maps can diverge from the department’s highest-impact opportunities. The proposed responses are temporary individual assignments, having a whole team contribute to another team’s goals, and forming a cross-functional tiger team; each shifts trade-offs among focus, maintenance, expertise, motivation, and management oversight. A month-long tiger team built the first Spin proof of concept, which led to a dedicated team, but no structure eliminates the tensions, so choices must be judged according to context.
Hubble: Our Tool for Encapsulating and Extending Security Tools
Shopify’s Trust team faced rising complexity in securing a distributed device fleet, including many tools, provisioning and patching work, varied networks, and the loss of in-person remediation. Hubble was built as a unified layer over mobile device management and other security systems, ingesting and standardizing their data while sending commands back with granular access controls and centralized auditing. IT staff use it for inventory, device management, and security, while employees can view device health and compliance, receive remediation guidance, and manage test-device or beta participation. The platform also serves as a standardized source for automation, and the post presents encapsulation and cross-team investment as ways to reduce administrative overhead, support proactive security, and limit risk.
How to Structure Your Data Team for Maximum Influence
Embedded data science leaders may support multiple product and business areas, creating competing work streams, specializations, and stakeholder relationships. The post compares swim lanes and stochastic process assignments against principles of efficiency, influence, stakeholder clarity, stability, growth, and flexibility. It recommends a hybrid “diamond defense” in which scientists are loosely assigned to zones for specialization while a bullpen, including the manager and an additional scientist, supplies reserve capacity. Resources can be redirected to overloaded areas, and team members can rotate into vacancies or cover leave. The framework is presented as combining clear ownership and subject-matter depth with the flexibility to handle change, though the post warns that excessive rotation can create volatility and turnover.
The Engineering Story Behind Flex Comp
Shopify’s Flex Comp lets employees allocate total rewards among base salary, Restricted Stock Units (RSUs), and Stock Options, with allocation changes during windows and no one-year equity cliffs. Building the system required a prioritized backlog, cross-functional teams, early API contracts, and privacy controls around its code and data. For sensitive analytics events, the team encrypted entire Kafka payloads with RSA, using 4096-bit keys for ordinary events and 6144-bit keys for larger payloads, while later identifying hybrid encryption as a better design. On the frontend, typed hooks, mocked data, and artificial delays allowed business logic and interface work to proceed in parallel while contracts evolved. Version one shipped to Shopify’s employees, with documented technical and design debt and future decisions retained for subsequent iterations.
Postgres WASM by Snaplet and Supabase
Supabase and Snaplet open-sourced postgres-wasm, a PostgreSQL server that runs inside a browser and supports browser persistence, pg_dump restoration, and logical replication with a remote database. Rather than compiling PostgreSQL directly to WebAssembly, the project packages a stripped-down Linux image with PostgreSQL in a Buildroot-based virtual machine, then runs it through v86 in the browser. PostgreSQL 14 startup failures were resolved by disabling v86 JIT compilation, while image optimization used a compressed 9P filesystem, asynchronous HTTPS loading, and page_poison=on to produce an initial state of about 12MB. Websocket proxying provides outbound connectivity and a forked proxy with nginx tunnels incoming connections to PostgreSQL. The project remains experimental because the download is currently about 30MB, but the post identifies offline data, testing, development environments, snapshots, and support as potential uses.
Lessons From Building iOS Widgets
Shopify’s widget team describes a third-generation redesign for iOS and Android after the original iOS widgets offered only two metrics and consumed substantial space. The widgets target merchants who need fresh shop analytics, using native SwiftUI on iOS because WidgetKit extensions have memory constraints, serialize SwiftUI view hierarchies to disk, and lacked official React Native support. Independent widget-side fetching uses shared session-management code and tokens; repeated update callbacks caused duplicate network requests and token-refresh races, so the team introduced a short-lived, widget-specific cache. Replacing existing widgets also exposed migration limits: changing widget kind identifiers could blank or remove installed widgets, leading Shopify to deprecate old widgets while releasing new ones and guide merchants through the transition. The team planned further customization and integration with iOS 16 features, including watch and lock-screen complications and live activities.
Four Approaches to Debugging Server-side WebAssembly
Server-side WebAssembly offers Shopify Functions low cold-start latency and robust security, but debugging remains difficult because step-debugger support is less refined. The post compares four approaches: compiling to a native architecture, using LLDB or GDB with Wasmtime, using a dedicated WebAssembly debugger, and using browser developer tools. Native debugging generally provides the best variable rendering and a familiar workflow, though it may require stubs for external calls and cannot reproduce WebAssembly-specific behavior. Wasmtime keeps execution closer to the target environment, while dedicated and browser-based tools offer WebAssembly-specific capabilities but have missing features or lower-fidelity variable views. The recommended choice depends on whether the issue reproduces natively, and browser-based tooling may become more accessible as it improves.
Focus On Behavior, Not State, for a More Maintainable Codebase
Shopify’s work adding pre-orders and “try before you buy” to its Purchase Options APIs exposed code that coupled selling plans too tightly to subscriptions, making future purchase models harder to add. The team reframed the domain around orders and behaviors, treating subscriptions as recurring orders and distinguishing recurring or deferred delivery and billing where those properties mattered. It introduced a wrapper around selling-plan Active Record objects, replaced those objects outside their defining component, and exposed methods such as whether deliveries are recurring while keeping policy details encapsulated. Because platform behavior and merchant reporting need different concepts, the team also added a translation table populated at order creation, with calculated or merchant-provided labels and optional trait records; the maintenance cost isolates reporting semantics from platform and data-team logic.
Building a Realtime Trello Board with Supabase and Angular
The tutorial builds a Trello-style project-management board with Supabase, Angular, and Tailwind rather than stopping at a small “hello world” example. It defines boards, lists, cards, users, and user_boards in Postgres SQL, adds magic-link authentication, and uses Row Level Security policies plus helper functions and triggers to control access and maintain relationships. Angular components and services use the Supabase JavaScript client, while realtime is enabled for cards and lists and exposed through an Observable that applies INSERT, UPDATE, and DELETE events to local board state. The resulting clone supports shared boards for invited, already registered users, and the article provides SQL and source code for running the project with a Supabase instance.
PostgREST v10: EXPLAIN and Improved Relationship Detection
PostgREST v10 adds HTTP and supabase-js access to PostgreSQL execution plans for reads, writes, and RPC requests, with the feature restricted to development-oriented use through db-plan-enabled. Plans can be returned as text or JSON, analyzed to show execution time and RLS filtering, and queried in verbose form for the pg_stat_statements query identifier and related settings. The release also improves relationship detection: unique or foreign-key primary keys produce one-to-one JSON objects, while computed relationships let SQL functions define efficient links for views and materialized views. Many-to-many detection now requires a join table’s primary key to include both foreign-key columns, a breaking change; the post says PostgREST 10 was not yet on the Supabase Platform and was planned for release there over the following month.
Realtime: Multiplayer Edition
Supabase announces the general availability of Realtime’s multiplayer features, Broadcast and Presence, alongside a new channel-based architecture and supabase-js support. Realtime originally used Postgres logical replication, rather than LISTEN/NOTIFY, to avoid the latter’s 8,000-byte payload limit and the latency of querying each change from the database. In the new design, Channels act as rooms that support ephemeral publisher/subscriber Broadcast messages, shared Presence state, and Postgres change events; Presence uses Phoenix Tracker, a delta-based CRDT, for eventually consistent synchronization and removes disconnected users automatically. Realtime now runs as a distributed Elixir and Phoenix cluster on Fly, with clients connecting to the nearest node and nodes communicating across a backbone network to improve connection speed, stability, and latency. Free-plan active projects and projects created from August 18 have access, while other projects are scheduled for migration over the following weeks.
How We Enable Two-Day Delivery in the Shopify Fulfillment Network
Shopify Fulfillment Network (SFN) enables merchants to offer two-day delivery across the United States by balancing inventory across Shopify-certified warehouses and choosing suitable carrier services. Shopify’s platform adds GraphQL APIs that let any third-party logistics provider upload pre-computed delivery dates, while storefront and checkout surfaces display them to buyers. SFN calculates schedules using inventory proximity, product weight, dimensions and hazard constraints, warehouse calendars and capacity, carrier pickups, closures, capacity limits, and time-in-transit data. It groups SKUs with shared inventory locations and characteristics into reusable schedules, maps each SKU to one, and supports direct delivery-date lookups in under 100 milliseconds; the selected date is returned to SFN for fulfillment routing.
How we store and process millions of orders daily
The Grab Order Platform processes millions of food and mart transactions daily, requiring high throughput, fault tolerance, and reduced cloud costs across transactional and analytical workloads. To meet these demands, the engineering team decoupled their database architecture by using Amazon DynamoDB for critical OLTP queries and MySQL RDS for historical OLAP queries. DynamoDB handles online order lifecycles with strong consistency, utilizing sparse Global Secondary Indexes for ongoing orders and TTL configurations to limit storage growth. Updates propagate asynchronously to MySQL RDS through a Kafka ingestion pipeline backed by Amazon SQS retries and timestamp-based version checks. This dual-database approach isolated core transaction availability from analytical queries and delivered significant cloud cost savings.
Mastering React’s Stable Values
React stable values are values, often returned by hooks, that retain the same value across renders, and the post explains why their identity matters in effects, callbacks, memoization, and component rendering. useState and useReducer update functions and useRef objects are stable, while array literals, object literals, and anonymous functions are recreated on each render; useCallback and useMemo can produce stable values under appropriate dependencies. Unstable dependencies prevent useEffect behavior from remaining limited to intended changes and stop useCallback and useMemo cached results from being reused, while an unstable callback passed to useFocusEffect causes repeated unsubscribe and resubscribe cycles. The post also contrasts invoking an inline helper with rendering it as a component, explains why the latter can rebuild a React Native subtree on every render, and recommends defining components outside other components and using React.memo or callbacks judiciously.
Implementing "seen by" functionality with Postgres
Tracking unique post views within Postgres presents trade-offs between counter accuracy, row bloat, and concurrent write performance. To evaluate options under an architecture constraint preventing external dependencies, a benchmark suite generated synthetic users, skewed post distributions, and replayed view actions. The benchmark tested four implementations: a naive counter column, an hstore key-value approach, an association table, and HyperLogLog (HLL). Benchmark results demonstrated that simple-hstore achieved the lowest average latency among deduplicating approaches at 2.15 milliseconds, closely followed by HLL at 2.16 milliseconds. Despite hstore's raw performance in prototypes, HLL is recommended for production because it circumvents row bloat as view counts grow while avoiding expensive row counts.
Typeahead Search at Nextdoor
Nextdoor built a proximity-based autocomplete service to power typeahead search and mention features across its hyperlocal platform for hundreds of millions of entities, including users and businesses. The system shards geographic data using Uber's open-source H3 geohashing library and stores prefix indexes in memory using Redis sorted sets. By adopting a Command Query Responsibility Segregation architecture, ingestion writes are processed on Redis primary nodes and replicated to read-only search nodes with under 10 milliseconds of replication lag. Dedicated APIs handle indexing, typeahead lookups, and ranking before returning hydrated results. Operating since August 2021, the service processes hundreds of millions of monthly typeahead queries while maintaining a P95 search latency below 30 milliseconds.
Shopify and Open Source: A Mutually Beneficial Relationship
Shopify describes its relationship with Ruby, Rails, and the open-source community as mutually beneficial, linking the company’s success to its Rails foundation and its engineering culture to the Rails Doctrine. It frames a 100-year company ambition around making Ruby and Rails viable for large teams, mature codebases, millions of lines of code, and billions of web requests through production performance, architecture guidance, upgrades, tooling, and selective typing. To support that goal, Shopify contributes projects and initiatives spanning developer tooling, gradual typing, VS Code support, automated upgrades, YJIT, Ruby performance work, TruffleRuby collaboration, $500,000 in research funding, Bundler and RubyGems, and RubyGems.org security. The post argues that company participation improves engineering skills, application longevity, and influence over foundational software, while describing contributors who range from occasional participants to full-time maintainers.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.