API Versioning as a Translation Layer
API versioning as an architecture: requests enter through a version header, pass through a chain of reversible version changes, and reach controllers that only ever see the latest model.
Loading…
Topic
Architecture
830 posts about Architecture. Every summary links to the original.
- 830 posts
- 50 companies
- 31 explainers
See the idea
Architecture explainers
BIAN as a Contract Layer: Modernizing Core Banking One Domain at a Time
BIAN gives banks a shared vocabulary and service-domain contracts that sit between business concepts and the systems implementing them. See how domains, contracts, and governed access stack into one architecture.
Argo CD: The Reconciliation Loop Between Git and Your Cluster
Argo CD is a GitOps controller that treats Git as the desired state and continuously reconciles it against what actually runs in each cluster. It maps the parts of that loop and why pull-based agents change its scale.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
Supabase Launches NFT Marketplace
Non-fungible tokens frequently suffer from the copy-paste problem, where public URLs allow unauthorized users to download underlying media directly from web hosts or IPFS. To address this issue, BuyMeth proposes combining public blurhash thumbnails with full image files encrypted under the active owner's cryptographic key and hosted on IPFS. Completed sales trigger Metamask to re-encrypt the file with the buyer's public key, followed by a one-week escrow challenge window where automated verification matches decrypted image hashes against public thumbnails. Ongoing royalty distributions to previous owners provide a financial mechanism to disincentivize leaking unencrypted original image files. The announced platform serves as an April Fools joke rather than an active Supabase product release.
Storage is now available in Supabase
Supabase launched Storage, adding a scalable object store to its existing Postgres, authentication, and API services. Existing open-source storage servers like Ceph, Swift, Minio, and Zenko were evaluated but rejected due to auth incompatibilities and external dependencies like etcd, MongoDB, and Kafka. Consequently, the team implemented a custom Storage API server built with Fastify and TypeScript behind the Kong gateway. Object metadata and access control reside directly in Postgres, leveraging Row Level Security policies written in SQL rather than a proprietary domain-specific language. Objects stream directly to managed backends like AWS S3 using Node streams with minimal in-memory buffering, accompanied by a default one-hour Cache-Control header.
How to Build a Web App with and without Rails Libraries
This tutorial builds Mirth, a Ruby web application that lets users view displayed data and enter new data that persists, using a birthday tracker as its example. It first constructs the application with core Ruby libraries, explaining TCP sockets, HTTP requests and responses, persistent storage, and the need for a web server interface such as Rack. It then replaces that lower-level code with Rails libraries, including Action Controller, Action Dispatch, Active Record, and Action View, while using ERB templates and a configured view path. The tutorial shows how routing, database access, request handling, and HTML generation fit together, with the application available at localhost:1337/birthdays after the Rails-library version is run. Its conclusion is that building from scratch clarifies the implementation details Rails hides, while Rails remains the practical choice for web development.
Remove Circular Dependencies by Using Dependency Injection and the Repository Pattern in Ruby
An internal Ruby pricing gem used by Shopify Core and Storefront Renderer needed consumer-owned data while retaining shared pricing knowledge, creating a circular dependency. The proposed design puts calculation logic and shared domain types in a stateless gem, while each consumer implements a repository contract for retrieving and returning the required data. Constructor injection passes a PricingRepositoryInterface implementation into PricingEngine::Engine, and Sorbet interfaces and function signatures enforce implemented methods and expected return types. Testing separates gem-isolated tests using repository mocks, consumer unit tests for repository behavior, and integration tests confirming the gem works within each consumer. The result, according to the post, is removal of the circular dependency and a typed contract that makes consumers responsible for data access.
Cracking PostgreSQL Interview Questions
The guide prepares candidates for PostgreSQL interviews by concentrating on transactional SQL rather than mixing it with analytical topics such as window functions, ranking, and aggregates. It covers relational modeling with foreign keys and join tables, PostgreSQL’s richer types, schemas, indexing strategies, and EXPLAIN ANALYZE for comparing estimated and actual query costs. It also addresses views, automatically updatable views and their limitations, database permissions, roles, PUBLIC grants, row-level security policies, and business logic implemented with PL/pgSQL functions and triggers. The guide recommends thinking in sets with SQL operations such as JOIN and UNION, using common table expressions for readable or recursive queries, and practicing these topics to build a strong foundation for PostgreSQL interview answers.
Using Betas to Deploy New Features Safely
Continuous deployment makes changing a running system risky, especially for a platform serving more than a million merchants and changing code multiple times daily. The post presents beta flags as a control mechanism: per-Subject BetaFlags support explicit activation, while BetaRollouts use a percentage and a digest-based modulo calculation to assign a stable, growing subset as rollout increases. It then places a higher-level Feature abstraction above those primitives, allowing direct targeting, percentage rollout, per-Subject opt-outs, and a 100% kill switch that can halt a feature without deleting thousands of database records. The guidance also covers testing permutations, notifying owners through BetaIdentifier metadata, and removing stale flags after rollout. These patterns provide operational control and safer deployment, while requiring substantial implementation and ongoing cleanup.
Customer Support Workforce Routing
Grab replaced its third-party customer support routing software with an in-house workforce routing system for Livechat to gain better priority controls, bespoke configurations, and deeper analytics. The platform separates requests into distinct priority and business queues, using parallel workers that spend varied time slices dequeuing higher-priority issues like safety concerns. To prevent request starvation, workers operate out of sync across queue priority levels while dynamic queue limits cap incoming volume based on agent availability and performance. The system routes requests through an intermediate Agent Group layer, calculating eligibility scores from proficiency and concurrency metrics while managing per-agent locks to prevent over-allocation.
Serving Driver-partners Data at Scale Using Mirror Cache
Grab's Drivers Data service handles up to 10,000 requests per second during peak hours to supply driver information across backend microservices. The original setup used MySQL with Redis and standalone in-memory local caches, but yielded a low 25% local cache hit rate due to traffic patterns characterized by high burst frequency for individual drivers alongside redundant database calls across nodes. To solve this, the team developed Mirror Cache, an in-memory caching system that pairs Dgraph's Ristretto library with an asynchronous gRPC replication layer to mirror updates across cluster nodes. The replicator batches updates within the same AWS availability zone and forwards data to single nodes across zones to minimize transfer overhead. Production deployment increased the in-memory cache hit rate to approximately 75% and reduced direct MySQL queries by 5%.
The GrabMart Journey
During the COVID-19 pandemic, Grab accelerated the rollout of its GrabMart on-demand grocery delivery service to meet surging regional demand. Early pilot testing within the existing GrabFood interface revealed that grocery shopping fundamentally diverges from meal ordering, as users search at an item level rather than browsing merchants. To support grocery workflows, the team restructured the user experience with item and store categories, one-click add-to-cart functionality, and scheduled delivery options. Engineering teams simultaneously updated core infrastructure across order management, driver allocation systems, and merchant Partner APIs to handle specialized mart bookings. Rollouts were validated across two main phases using the internal Grab Early Access program to catch issues before regional release.
Trident - Real-time Event Processing at Scale
Trident serves as Grab's internal real-time event-processing and workflow automation engine, driving user campaigns, rewards, and notifications across multiple business lines. To handle peak loads exceeding 2,000 events per second without duplicate execution, the system consumes decoupled Kafka streams and enforces exactly-once semantics using Redis and MySQL deduplication checks. Processing efficiency relies on server autoscaling aligned with Kafka partition counts, combined with dynamic goroutine allocation per consumer. To minimize rule evaluation overhead, Trident indexes active campaigns into an in-memory hash map by event type, cutting processing time by at least 90%. Furthermore, condition evaluation is optimized through lazy loading and a weighted sorting algorithm that checks low-cost in-memory data prior to executing expensive database queries or external service calls.
Pharos - Searching Nearby Drivers on Road Network at Scale
Ride-hailing allocation requires identifying nearby drivers based on actual routing distance and estimated time of arrival rather than straight-line haversine distance. Grab developed Pharos, a distributed in-memory spatial microservice that evaluates road networks to match fast-moving drivers with pick-up requests. The system partitions OpenStreetMap road graphs by city and vehicle vertical while indexing driver positions and edge-based nodes using Adaptive Radix Trees. To locate candidates, Pharos projects driver coordinates onto road segments as phantom nodes and runs Incremental Network Expansion to compute isochrone reaches. Read-write concurrency is managed by switching tree root snapshots, allowing concurrent reads while updates produce new roots for subsequent lookups.
New VMs: more RAM, extra CPU, and a dollar menu
Fly.io introduces new virtual-machine configurations after developers shifted attention from CPU-intensive workloads such as image processing, machine learning predictions, and video transcoding toward databases, which need higher RAM-to-CPU ratios. Shared-cpu VMs offer up to 2GB of RAM, while dedicated-cpu VMs reach 64GB across configurations with one, two, four, or eight dedicated CPUs. Shared-cpu-1x pricing starts at $1.94 per month for 256MB and reaches $10.70 for 2GB; dedicated-cpu-1x with 2GB costs $31 per month. The pricing model combines dedicated hardware with shared CPU capacity, averaging 12 shared VMs per CPU and using cgroups priorities and relocation to manage contention. Persistent storage remains unavailable, so the post says ephemeral VMs are unsuitable for database applications and that Fly.io should offer storage.
A World Rendered Beautifully: The Making of the BFCM 3D Data Visualization
The 2020 Black Friday Cyber Monday Globe and a related Live View visualization had to represent orders from one million merchants within two months while remaining realistic and performant. The team used a reusable layer architecture with three.js, physically based rendering, 32-bit EXR environment maps, and custom materials and shaders for the Earth and carbon-offset bubble effects. To display thousands of orders, it replaced arc-heavy rendering with particle-based visuals built from Points, BufferGeometry attributes, and ShaderMaterial. An automatic optimizer monitors frame rate and, when it stays below 55 fps for more than two seconds, lowers canvas pixel ratio to 1x and reduces environment-map resolution, helping low-power devices return to 60 fps. The resulting implementation became an internal library for future globes.
Democratising Fare Storage at Scale Using Event Sourcing
Grab's legacy system stored booking and fare details in a single relational table, creating a bloated booking entity that tracked only the latest fare state and hindered rapid feature iteration. To resolve scalability, stability, and debugging challenges across millions of daily bookings, the team developed Fare Storage using the Event Sourcing pattern. The new architecture persists all fare modification events chronologically in DynamoDB, backed by a cache for eventually consistent reads and message streaming for downstream processing. The platform employs optimistic locking with versioning to manage concurrent updates, enforces idempotency through client-generated transaction UUIDs, and delegates metadata serialization to an SDK to prevent storage API changes.
How to Introduce Composite Primary Keys in Rails
Shopify addressed inefficient data access in its multi-tenant Rails application, where auto-incrementing primary keys interleaved records from many shops even though queries usually targeted one shop. InnoDB stores rows in B+ trees and loads pages into a buffer pool, so the team changed orders and other suitable tables to composite primary keys such as [shop_id, order_id] while keeping an auto-incrementing id secondary key and configuring Active Record to use id. Supporting Ghostferry, migration tooling, and data extraction systems also had to accommodate the schema change, with unique secondary constraints avoided in some cases because LHM migrations could deadlock. On the most queried table, common queries improved 5–6x, median buffer pool reads fell from 1.8 to 1.2, and elapsed database time dropped roughly one hour per day per shard, while inserts became about 10x slower.
Building Mental Models of Ideas That Don’t Change
Amid a constant stream of frameworks, languages, and platforms, the post argues that learning enduring principles is more useful than trying to absorb every new tool. It presents mental models as a checklist for prioritizing concepts, recording what matters, and making more informed decisions across engineering and management. Engineering examples include alerting on silent failures, queuing background jobs for faster responses and retries, using caching and denormalization for read-heavy workloads, and sharding or deliberate design choices for write-heavy systems. The post also recommends horizontal scaling for long-term growth, then turns to practices such as aligning incentives, clarifying product purpose, building rapport before feedback, and operating a few months ahead. The stated aim is to use these models to make more informed decisions rather than chase an ever-expanding list of new technologies.
Images as Code: Representing Localized and Evolving Products on Marketing Pages
Shopify needed product visuals on more than 35 localized domains to stay accurate as features, currencies, languages, and product UI changed. An SVG experiment made translated text difficult to manage because SVG lacks word wrapping, while export settings, effects such as Gaussian blur, and repeated translation hooks created maintenance and performance problems. The replacement represented interfaces in HTML and JavaScript through ScaleContentAsImage, which wraps fixed-width content, computes available space, applies a CSS transform scale, preserves layout with a spacer, absolutely positions the transformed element, and updates calculations on resize. The team concluded HTML/JavaScript was more maintainable and flexible for design, documentation, animations, and localization, while noting remaining work around lazy-loaded images, reusable-visual documentation, and future CSS Houdini possibilities.
Enforcing Modularity in Rails Apps with Packwerk
Large Rails monoliths can develop high coupling, low cohesion, cross-component calls, shared Active Record models, and classes that know too much because Ruby and Rails provide limited boundary enforcement. Shopify created Packwerk, an open-source static analysis tool that groups Ruby files into packages and enforces dependency and privacy boundaries, including controlled public APIs for constants. It reports the violation type and location with actionable next steps, integrates with CI, and can run locally; installation begins by adding the gem and running packwerk init. Because Ruby static analysis is complex, Packwerk ignores constants that are not autoloaded, reducing false positives at the cost of false negatives. The excerpt says it runs in six Shopify Rails applications, with 48 packages and 30 boundary enforcements in the core codebase, while adoption also prompted work on dependency inversion.
Under Deconstruction: The State of Shopify’s Monolith
Shopify describes the ongoing effort to modularize its core Ruby on Rails monolith, which contains more than 2.8 million lines of Ruby and 500,000 commits. The initiative organizes code into independently owned components intended to narrow developer focus, reduce affected test suites, preserve contracts, and clarify operational ownership, while recognizing that large-scale refactoring is also a people problem. The work has already improved exception triage, distributed codebase chores, and design awareness, while its current practices emphasize grassroots participation, tooling, targeted reviews, and a holistic architectural view. Packwerk restricts dependencies for about a third of the 37 main-monolith components. Shopify is pursuing a cleaner dependency graph, componentized Rails applications by default, and isolated component tests, while reserving service extraction for cases such as storefront rendering and credit-card vaulting.
Tophatting in React Native
Shopify’s tophatting process manually verifies coworkers’ changes before pull requests are approved, but React Native reviews previously required saving work, switching branches, rebuilding, and loading the app. The new tool stores compressed JavaScript bundles and their assets in cloud storage when Shopify Build’s CI pipeline processes a React Native pull request. A backend service records metadata such as the app ID, commit SHA, and branch, generates signed URLs, and posts a pull-request comment with a QR code, bundle ID, and download link. A React Native component library handles deep links through a separate Tophat screen, confirms bundle metadata, downloads and decompresses the selected bundle, saves metadata locally, and restarts the app with it. The first iteration lets developers load peer changes within seconds, and almost all Shopify React Native projects now use the tool.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.