API Versioning as a Translation Layer
API versioning as an architecture: requests enter through a version header, pass through a chain of reversible version changes, and reach controllers that only ever see the latest model.
Loading…
Topic
Architecture
830 posts about Architecture. Every summary links to the original.
- 830 posts
- 50 companies
- 31 explainers
See the idea
Architecture explainers
BIAN as a Contract Layer: Modernizing Core Banking One Domain at a Time
BIAN gives banks a shared vocabulary and service-domain contracts that sit between business concepts and the systems implementing them. See how domains, contracts, and governed access stack into one architecture.
Argo CD: The Reconciliation Loop Between Git and Your Cluster
Argo CD is a GitOps controller that treats Git as the desired state and continuously reconciles it against what actually runs in each cluster. It maps the parts of that loop and why pull-based agents change its scale.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
Being a Principal Engineer at Grab
Grab's rapid growth resulted in roughly 350 microservices powering its superapp, creating the need for defined individual contributor career milestones. At Grab, a principal engineer oversees the architecture of an entire Tech Family, a sub-organisation containing over 50 engineers and 20 or more microservices. Responsibilities include translating broad, ambiguous problems into concrete projects, managing technical debt, and aligning multiple engineering teams across global R&D centres. The role demands continuous technical leadership through RFC design reviews, cross-functional communication, mentorship, and self-directed prioritization alongside engineering leadership. Ultimately, principal engineers amplify engineering quality and operational stability without directly managing people.
No More Forgetting to Input ERP Charges - Hello Automated ERP!
Grab launched an automated Electronic Road Pricing (ERP) fare calculation feature in Singapore to eliminate the need for driver-partners to manually track gantries and enter toll charges. Because Singapore gantries frequently adjust fares based on time and road conditions, manual entry often caused driver errors and revenue loss. Grab solved this by mapping precise geographical coordinates for every toll gate using satellite imagery and open data, matching frequent driver GPS pings against road layers and gantry locations. The engineering and operations teams also built an internal ERP Workflow tool to map ride trajectories and resolve driver dispute feedback within an average of one day. Following its rollout in Singapore, Grab began testing and planning regional expansion to Indonesia, Thailand, Malaysia, and the Philippines.
How We Built a Logging Stack at Grab
Grab needed a scalable logging platform to replace slow, fragmented systems that hindered debugging across their growing service fleet. Generating 25TB of daily logs, the team built a horizontally scalable Elasticsearch cluster configured via Ansible and monitored with Datadog. Although the initial proof of concept assigned all node roles (ingest, coordinator, master, and data) to every machine, operating at scale introduced major challenges with JVM heap exhaustion and cluster stability. The team resolved memory pressure and performance bottlenecks by tuning circuit breakers, lowering field data cache limits, adjusting shard allocations based on segment memory, and disabling translog compression during shard transfers.
How Heroku Operates its Multi-Tenant Apache Kafka Services
Heroku’s Apache Kafka service supports thousands of developers and millions of transactions, including customers sharing multi-tenant clusters. The post explains how those clusters provide security, predictable performance, availability, and production-like topology at lower cost than single-tenant plans. Multi-tenant and single-tenant offerings use the same numbers of brokers and ZooKeeper nodes, while Kafka ACLs, per-tenant resource namespaces, and producer/consumer byte quotas isolate customers and limit resource contention. Heroku provisions purchased resources, expands storage at 80% of quota, and uses higher defaults for partitions, replicas, and in-sync replicas; exceeding capacity can eventually throttle throughput without stopping the cluster. Automated tests model 50 users with varied traffic, exercise throughput and partition configurations, and simulate broker failures and restarts to identify operational limits and adjust the infrastructure before customer impact.
Componentizing Shopify’s Tax Engine
Shopify’s tax engine grew from a simple system into difficult code as tax rules and ownership spread across Cart, Checkout, and Order. The team first learned the surrounding code and tax domain with help from a tax subject matter expert, then mapped dependencies using regular expressions, scripts, manual investigation, and tax-related bug work. They moved tax source files into a dedicated component behind explicit TaxesRequestSchema and TaxesReponseSchema interfaces, with simple-type request and response schemas, protected boundaries, and expanded unit, integration, and end-to-end tests. During release, the old and new engines calculated taxes concurrently so discrepancies could be measured; after iterative fixes, Shopify rolled the component out gradually from a small group of shops to all shops over several days, with no downtime or merchant impact reported.
Making Grab’s Everyday App Super
Grab manages an expanding superapp ecosystem comprising ride-hailing, food delivery, payments, and partner content surfaced through the Grab Feed. As content volume grows, the platform risks overwhelming users with irrelevant information. To address this, Grab built a recommendation engine that ranks cards using signals across user profiles, content metadata, and contextual factors such as time and location. The system employs multiple recommendation strategies—including popularity metrics, user favorites, collaborative filtering, habitual patterns, and cross-platform deep embeddings—which are selected or aggregated. Recommendation quality is evaluated via offline metrics like Recall@K and NDCG alongside online engagement experiments.
Catwalk: Serving Machine Learning Models at Scale
As machine learning adoption expanded at Grab, individual teams created fragmented model serving solutions that duplicated engineering effort and required data scientists to handle underlying infrastructure. To resolve these inefficiencies, Grab developed Catwalk, a self-service machine learning model serving platform. The system runs TensorFlow Serving containers across a managed Kubernetes cluster integrated with Grab's observability stack. Data scientists deploy or update models simply by saving files using the tf.saved_model API to dedicated Amazon S3 buckets, while Kubernetes automates orchestration, ingress routing, and pod autoscaling. Catwalk abstracts server management away from data scientists, shortens deployment timelines, and provides high availability during model version rollouts.
Implementing Android POS Receipt Printing on Shopify
Shopify’s Android POS lacked printed-receipt support until March 2019, leaving merchants who needed it to use iPads. The design targets a pragmatic, adaptive, composable, maintainable, and highly testable API, while handling more than 8 billion possible receipt content combinations across discounts, tips, transactions, gift cards, refunds, payment methods, and regional requirements. Its six-step pipeline aggregates models from SQLite or GraphQL, transforms them into a PrintableReceipt, defines view-based sections, creates printer-agnostic PrintRequest objects, renders section bitmaps at the required resolution, and passes them through the Hardware SDK to vendor-specific integrations. The PrintableReceiptComposer contains most business logic, with unit tests covering features and edge cases, while screenshot tests compare rendered output pixel by pixel in CI. This componentized separation isolates printer support from receipt generation and allows new printer models to be added in a couple of hours.
React Native in GrabPay
Following the release of the GrabPay Merchant App, Grab adopted React Native inside the Grab Passenger app to maintain a single cross-platform codebase across iOS and Android. Integrating the framework into native applications required establishing bridge communication guidelines and incorporating React Native modules into the Grablet architecture. To streamline network communication, API calls were migrated from axios to native bridges returning promises, which eliminated the need to pass access tokens into JavaScript. The team also established a shared internal library of approximately 20 UI components alongside Redux for state management and react-navigation for routing. Modules like BillPay and Transaction History successfully launched across Southeast Asia while maintaining the performance and feel of native software.
Connecting the Invisibles to Design Seamless Experiences
Service design operates as connective tissue within complex product ecosystems by bridging digital touchpoints, physical operations, and backstage technical workflows. At Grab, placing an order on GrabFood requires coordinating driver allocation, customer support paths, and long-term data storage rather than simply transmitting information to merchants. Focusing exclusively on singular features risks breaking broader network dependencies when modifications ripple into other operational systems. Grab addresses these interdependencies through participatory design processes and visual mapping across cross-functional teams. This holistic framework evaluates whether systemic issues, such as inaccurate restaurant operating hours, are best resolved through in-app feature changes or operational adjustments.
Designing Resilient Systems Beyond Retries (Part 3): Architecture Patterns and Chaos Engineering
Building resilient systems requires architectural safeguards and proactive testing beyond basic retries and circuit breakers. Architectural patterns such as idempotency keys enable safe retries without creating inconsistent state during failures. Asynchronous responses and deferrable work isolate services from downstream dependency latency and errors, though they can conflict with the fail-fast principle. To validate system behavior under stress, chaos engineering introduces intentional failures in production to test hypotheses against a defined steady state. Selectively adopting complementary patterns reduces failure points while avoiding unnecessary architectural complexity.
Designing Resilient Systems Beyond Retries (Part 2): Bulkheading, Load Balancing, and Fallbacks
Software systems require mechanisms beyond retries to maintain resilience during downstream outages and high traffic. Bulkheading isolates failures across infrastructure, processes, thread pools, and connection limits, preventing a single failing component from degrading an entire system. Load balancing distributes traffic across backend pools via proxies, client-side libraries, lookaside services, or sidecars, often pairing with health checks to eliminate single points of failure. When operations fail unrecoverably, fallback strategies like silent failures, local defaults, stale cache reads, and dedicated backup services enable graceful degradation. Organizations like Grab implement these approaches using internal client-side load balancers backed by etcd, cache fallbacks in microservice frameworks, and redundant core backup services.
Designing Resilient Systems Beyond Retries (Part 1): Rate-Limiting
Distributed systems that rely exclusively on retries and circuit breakers face severe failure risks, including retry storms and reliance on client-side configuration accuracy. Implementing server-side rate limiting serves as a critical defensive layer to safeguard services across evolving architectures. Throttling thresholds can be layered across per-client, per-endpoint, and server-wide granularities using algorithms such as leaky bucket or sliding windows. While local instance-level limits fail when downstream bottlenecks like databases saturate under horizontal scaling, global rate limiting coordinates traffic enforcement across entire service pools. Centralized rate limiters require asynchronous communication and fallback mechanisms to avoid becoming single points of failure or adding request path latency.
Context Deadlines and How to Set Them
Microservice architectures operating under heavy network traffic require robust timeout handling to prevent slow or failing dependencies from causing cascading failures across services. Naive, static timeout configurations across a call chain often cause downstream components to waste compute effort on requests that upstream callers have already abandoned. To establish predictable timeout thresholds, engineers can align limits with service-level latency percentiles, pairing P99 limits with median latency retry allowances. Go's context package improves upon static network timeouts by propagating request-scoped deadlines and cancellation signals across service boundaries. This distributed context ensures downstream servers recognize remaining time budgets and terminate unneeded processing immediately when parent deadlines expire or callers manually cancel requests.
Structured Logging: The Best Friend You’ll Want When Things Go Wrong
Grab redesigned its backend logging approach to address mounting vendor costs, query language limitations, and debugging difficulties across a growing microservices ecosystem. Most services previously emitted syslog-style key-value logs almost entirely at the INFO level, which made volume reduction difficult and lacked causal ordering and automated correlation. The engineering team migrated to a self-managed Elastic stack backend and built a structured logging library in Go from the ground up. This framework introduces dynamic log-level adjustment at runtime, automatic trace-based log correlation via Grab-Kit, and the Common Grab Log Schema to enforce consistent JSON formatting without Elasticsearch indexing conflicts.
Deconstructing the Monolith - Shopify
Shopify describes why its decade-old Ruby on Rails monolith became difficult to change as the codebase and development organization grew. High coupling and absent boundaries made innocuous changes trigger unrelated test failures, slowed CI, and forced developers to absorb excessive domain context, while microservices would add deployment, infrastructure, network, and coordination costs. The chosen path was a modular monolith: retain one codebase and deployment unit while separating domains through strictly enforced component boundaries. Shopify’s Componentization effort was informed by a developer survey and uses Wedge to score components and report violations, including cross-component associations and calls that bypass explicitly public interfaces. Boundary enforcement at runtime remains a future goal, but dependency isolation has already enabled replacing a legacy tax engine with a new tax calculation system.
Unifying Our GraphQL Design Patterns and Best Practices with Tutorials
Shopify describes how its GraphQL work grew from an early experiment for mobile APIs into contributions from hundreds of developers across teams and offices, including the Admin and Storefront APIs. As adoption expanded, sparse Markdown guidance left context inconsistently shared, while design guidelines were mixed with code-reference documentation and became difficult to extend for complex cases. The API Patterns Team, created in May 2017, addressed this by focusing on horizontal concerns, identifying common patterns, and documenting solutions for developers working across API domains. This work produced a living, in-depth design tutorial that explains Shopify’s GraphQL design philosophy through a practical example; its central rule is to understand the business domain and design the API to reflect it rather than the application’s implementation.
A Lean and Scalable Data Pipeline to Capture Large Scale Events and Support Experimentation Platform
Controlled online experimentation across diverse product verticals requires tracking interactions across systems to prevent local optimizations from causing global degradation. Grab built a batch data pipeline to capture, ingest, and process petabytes of event data to support its experimentation platform and analytics stakeholders. The architecture loads ingested event data from Amazon S3, transforms and sorts it, and writes partitioned output back to S3 with metadata registered in Apache Hive. Using Apache Spark on AWS Elastic MapReduce with Apache Airflow for orchestration, the system handles roughly 400,000 incoming events per second. The data is partitioned by event type and ingestion time and stored in Apache ORC format to streamline query workloads and reduce retrieval overhead.
Building a Service-oriented Architecture with Rails and Kafka
Service-oriented Rails applications can use Kafka to move high-volume events between producers and consumers without coupling downstream availability to an upstream request. Kafka persists events for a configured period, lets consumers manage offsets, and distributes replicated data across broker partitions, while ordering is guaranteed within a partition rather than across a topic. The post shows a Rails setup using the ruby-kafka gem, an asynchronous producer, JSON payloads, topics, and partition keys, then applies the event stream to order creation, email, and fulfillment workflows. It also addresses operational concerns including consumer scaling, lag metrics, duplicate delivery, idempotent processing with UPSERTs or event identifiers, and schema evolution, contrasting JSON’s accessibility with Avro’s stronger schema support.
Designing Resilient Systems: Circuit Breakers or Retries? (Part 2)
Retries enable software systems to recover from transient upstream failures by automatically repeating unsuccessful requests. While retrying increases the chance of request completion across multi-host setups, it consumes additional CPU and time without inherently tracking host health. Applications must selectively retry errors with a likelihood of success, such as 500 and 503 status codes, while avoiding client-side failures like 400 or 401. To manage distributed systems safely, retries require idempotent operations or cryptographic nonces, along with backoff and jitter to prevent request stampedes. Tuning retry counts, timeouts, and delays is critical to cap the worst-case consumer response time.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.