---
title: "Architecture"
description: "834 posts about Architecture, summarised, each linking to the original."
---

# Architecture
> 834 posts about Architecture, summarised, each linking to the original.

## Articles

### [Unifying Our GraphQL Design Patterns and Best Practices with Tutorials](https://yomu.fyi/post/unifying-our-graphql-design-patterns-and-best-practices-with-tutorials.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Feb 13, 2019
- Type: Tutorial

Shopify describes how its GraphQL work grew from an early experiment for mobile APIs into contributions from hundreds of developers across teams and offices, including the Admin and Storefront APIs. As adoption expanded, sparse Markdown guidance left context inconsistently shared, while design guidelines were mixed with code-reference documentation and became difficult to extend for complex cases. The API Patterns Team, created in May 2017, addressed this by focusing on horizontal concerns, identifying common patterns, and documenting solutions for developers working across API domains. This work produced a living, in-depth design tutorial that explains Shopify’s GraphQL design philosophy through a practical example; its central rule is to understand the business domain and design the API to reflect it rather than the application’s implementation.


### [A Lean and Scalable Data Pipeline to Capture Large Scale Events and Support Experimentation Platform](https://yomu.fyi/post/a-lean-and-scalable-data-pipeline-to-capture-large-scale-events-and-su.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Oscar Cassetti
- Published: Jan 16, 2019
- Type: Problem & solution

Controlled online experimentation across diverse product verticals requires tracking interactions across systems to prevent local optimizations from causing global degradation. Grab built a batch data pipeline to capture, ingest, and process petabytes of event data to support its experimentation platform and analytics stakeholders. The architecture loads ingested event data from Amazon S3, transforms and sorts it, and writes partitioned output back to S3 with metadata registered in Apache Hive. Using Apache Spark on AWS Elastic MapReduce with Apache Airflow for orchestration, the system handles roughly 400,000 incoming events per second. The data is partitioned by event type and ingestion time and stored in Apache ORC format to streamline query workloads and reduce retrieval overhead.


### [Building a Service-oriented Architecture with Rails and Kafka](https://yomu.fyi/post/building-a-service-oriented-architecture-with-rails-and-kafka.md)
- Company: [Heroku](https://yomu.fyi/company/heroku.md)
- Author: Stella Cotton
- Published: Jan 14, 2019
- Type: Tutorial

Service-oriented Rails applications can use Kafka to move high-volume events between producers and consumers without coupling downstream availability to an upstream request. Kafka persists events for a configured period, lets consumers manage offsets, and distributes replicated data across broker partitions, while ordering is guaranteed within a partition rather than across a topic. The post shows a Rails setup using the ruby-kafka gem, an asynchronous producer, JSON payloads, topics, and partition keys, then applies the event stream to order creation, email, and fulfillment workflows. It also addresses operational concerns including consumer scaling, lag metrics, duplicate delivery, idempotent processing with UPSERTs or event identifiers, and schema evolution, contrasting JSON’s accessibility with Avro’s stronger schema support.


### [Designing Resilient Systems: Circuit Breakers or Retries? (Part 2)](https://yomu.fyi/post/designing-resilient-systems-circuit-breakers-or-retries-part-2.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Corey Scott
- Published: Jan 8, 2019
- Type: Explainer

Retries enable software systems to recover from transient upstream failures by automatically repeating unsuccessful requests. While retrying increases the chance of request completion across multi-host setups, it consumes additional CPU and time without inherently tracking host health. Applications must selectively retry errors with a likelihood of success, such as 500 and 503 status codes, while avoiding client-side failures like 400 or 401. To manage distributed systems safely, retries require idempotent operations or cryptographic nonces, along with backoff and jitter to prevent request stampedes. Tuning retry counts, timeouts, and delays is critical to cap the worst-case consumer response time.


### [Querying Big Data in Real-time with Presto & Grab's TalariaDB](https://yomu.fyi/post/querying-big-data-in-real-time-with-presto-grab-s-talariadb.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Roman Atachiants
- Published: Jan 2, 2019
- Type: Problem & solution

Grab developed TalariaDB to support real-time SQL querying over high-velocity event streams while maintaining predictable sub-second latencies and low infrastructure costs. The distributed time-series store retains only the most recent hour of data and integrates directly with Presto via its PrestoThriftService interface. Internally, TalariaDB uses the Go-based Badger key-value store to maintain an in-memory key index of metric names and timestamps while mapping columnar event payloads directly to disk. Ingestion occurs by processing pre-partitioned event batches written to Amazon S3 via SQS notifications. By combining a zero-copy decoder with parallel split evaluation across gossiping cluster nodes, the architecture scales horizontally while serving millions of events per second.


### [Designing Resilient Systems: Circuit Breakers or Retries? (Part 1)](https://yomu.fyi/post/designing-resilient-systems-circuit-breakers-or-retries-part-1.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Corey Scott
- Published: Dec 21, 2018
- Type: Explainer

Distributed architectures frequently encounter upstream failures triggered by networking issues, system overloads, resource starvation, and invalid deployments. Implementing software circuit breakers interposes a monitoring mechanism between components to halt requests when failure thresholds are met, giving struggling upstream dependencies time to recover. Circuit breakers save CPU, memory, and network resources by failing fast or routing execution through defined fallbacks such as cached data, alternate services, or approximation algorithms. Grab utilizes Hystrix-Go to manage upstream interactions and configure key thresholds for concurrency, timeouts, and error ratios. This approach protects downstream consumers from cascading latency while insulating upstream resources from excess traffic.


### [Orchestrating Chaos Using Grab's Experimentation Platform](https://yomu.fyi/post/orchestrating-chaos-using-grab-s-experimentation-platform.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Roman Atachiants
- Published: Nov 23, 2018
- Type: Problem & solution

Grab operates hundreds of microservices where failures in non-critical components can cause outages in critical user flows if fallback mechanisms are improperly configured. To validate system resilience, Grab built Chaos ExP by layering a chaos engineering SDK and dedicated web UI on top of its existing Experimentation Platform. Integrated directly into the Grab-Kit server middleware, the framework intercepts incoming requests and evaluates whether to inject failures using local variable resolution. Supported failure primitives include latency, errors, panics, rate throttling, and resource leaks to test dependent services. Combining chaos testing with experimentation telemetry enables engineers to correlate injected infrastructure disruptions with business metric impacts.


### [Reliable and Scalable Feature Toggles and A/B Testing SDK at Grab](https://yomu.fyi/post/reliable-and-scalable-feature-toggles-and-a-b-testing-sdk-at-grab.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Roman Atachiants
- Published: Nov 2, 2018
- Type: Problem & solution

Grab previously managed experiments using custom service-level code and a toggling library that queried a shared Redis instance, creating latency risks and a single point of failure across backend microservices. To achieve reliable, sub-microsecond feature evaluations, the team designed a Go SDK that resolves rollouts and A/B tests entirely in memory without runtime network I/O. Backend services periodically poll JSON-defined configuration schemas stored in Amazon S3 through a Universal Configuration Manager. The SDK evaluates contextual attributes called facets locally and pushes decision telemetry asynchronously to an S3 and Presto data lake. This architecture allows engineering and product teams to gate deployments and run server-side experiments safely without service disruption.


### [Building Shopify POS for Android Using MVVM](https://yomu.fyi/post/building-shopify-pos-for-android-using-mvvm.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: Richa Khanna
- Published: Oct 31, 2018
- Type: Tutorial

Shopify’s Android Point of Sale app adopted Model-View-ViewModel (MVVM) to replace an inconsistent codebase containing both MVC and MVP. Because the roughly three-and-a-half-year-old app could not be migrated at once, the team used MVVM for all new screens and converted legacy screens during significant changes. In the architecture, Fragments and sub-views report UI actions through contracts, ViewModels fetch and mutate data, and LiveData delivers updates to active observers. ViewModelProvider retains ViewModel instances across Activity or Fragment configuration changes, while Models handle databases, networks, and caches. Shopify reports more consistent development, easier onboarding and testing, lifecycle-safe UI updates, and a codebase that is easier to modify, although some MVC and MVP screens remain.


### [Beyond Web and Worker: Evolution of the Modern Web App on Heroku](https://yomu.fyi/post/beyond-web-and-worker-evolution-of-the-modern-web-app-on-heroku.md)
- Company: [Heroku](https://yomu.fyi/company/heroku.md)
- Author: Chris Castle
- Published: Aug 14, 2018
- Type: Explainer

Over a decade, the standard Heroku web application has expanded from a web process, database, worker, and Redis queue into a broader architecture for web, mobile, data, and operational demands. The post connects this shift to snappier single-page experiences, native smartphone apps, APIs as integration surfaces, growing streaming data, increasing system complexity, and scale. Its proposed components include a content delivery network, API Gateway, Apache Kafka, object storage, data warehouses, and query or visualization tools, alongside service-oriented architecture and scalable-first design. The architecture separates functionality into smaller services for adaptability, while acknowledging that more components create more scaling and operational levers. It presents Heroku’s PaaS and managed services as a way to reduce that burden, then leaves deployment, communication, CI/CD, monitoring, and security as topics for follow-up posts.


### [How We Designed the Quotas Microservice to Prevent Resource Abuse](https://yomu.fyi/post/how-we-designed-the-quotas-microservice-to-prevent-resource-abuse.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Jim Zhan
- Published: Aug 10, 2018
- Type: Problem & solution

As Grab migrated from a monolith to hundreds of microservices, managing global rate limiting became essential to prevent cascading failures and resource exhaustion. To avoid putting a rate limiting service on the critical path of every API call, Grab built Quotas, an asynchronous rate limiting system. Client services use a lightweight SDK and middleware to read rate limiting decisions from local in-memory caches and stream usage metrics asynchronously via Apache Kafka. The Quotas service aggregates usage data locally, flushes stats to Redis periodically, and publishes updated rate limiting decisions back over Kafka topics. In production, Quotas successfully handles 200k peak transactions per second with decision enforcement delays capped at 200 milliseconds.


### [Building Grab’s Experimentation Platform](https://yomu.fyi/post/building-grab-s-experimentation-platform.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Abeesh Thomas
- Published: Jul 13, 2018
- Type: Problem & solution

Grab built its internal Experimentation Platform (ExP) to replace a manual, expensive testing process that required bespoke meetings, custom logging pipelines, and service modifications for each experiment. ExP provides a unified infrastructure featuring a centralized management UI, automated real-time data streaming to S3, and SDKs for Android, iOS, and Go. The platform leverages JSON-based experiment definitions delivered through dynamic configuration management, enabling client-side evaluation without costly network calls. It addresses marketplace network effects and inter-experiment interference through mechanisms such as geo-temporal segmentation and domain-layer models. The platform has scaled to run approximately 25 concurrent experiments while computing roughly 2,500 metrics and 50,000 experiment-metric combinations daily.


### [Rolling the Heroku Redis Fleet](https://yomu.fyi/post/rolling-the-heroku-redis-fleet.md)
- Company: [Heroku](https://yomu.fyi/company/heroku.md)
- Author: Camille Baldock
- Published: Jun 27, 2018
- Type: Explainer

Heroku proactively rolled its Redis fleet to a version not vulnerable to CVE-2018-11218 during an embargo, aiming to protect customers while limiting disruption and downtime. The operation relied on immutable infrastructure: engineers built, tested, reviewed, and released a new server image, then replaced High Availability standbys before scheduling customer maintenance. During each update, the team verified replication, broke the link so the standby could accept writes, redirected application config vars to the new primary, de-provisioned the old primary, and created a replacement standby. Automation controlled concurrency and limited replacement work to the on-call operator’s business hours, while customers received advance and completion emails. The fleet roll covered all Heroku Redis customers within five weeks of notification and finished one week before the public announcement; the process also applies to Heroku Postgres and Apache Kafka on Heroku.


### [Introducing Grab-Kit: Distributed Service Design at Grab](https://yomu.fyi/post/introducing-grab-kit-distributed-service-design-at-grab.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Karen Kue
- Published: Jun 8, 2018
- Type: Problem & solution

As Grab migrated from a monolith to microservices, maintaining consistency, coordination, and code quality across rapidly expanding teams became a major engineering challenge. To address this, the Developer Experience team built Grab-Kit, a Go framework that automates service scaffolding, code generation, and distributed system design patterns. The framework uses Protocol Buffer definition files as a single source of truth to generate data transfer objects, communication bindings, and standardized middleware for logging and profiling. Grab-Kit also features declarative metrics definitions that synchronize with the DataDog API to build and update service dashboards automatically. Adopting the framework reduced development time for creating new services by up to 70% in teams such as GrabFood while improving overall system stability.


### [Event Sourcing made Simple](https://yomu.fyi/post/event-sourcing-made-simple.md)
- Company: [Kickstarter](https://yomu.fyi/company/kickstarter.md)
- Author: Philippe Creux
- Published: Apr 20, 2018
- Type: Explainer

Event Sourcing stores application changes as immutable events, providing a history that can explain how current data came to be. The post introduces four components: Events record changes, Calculators update Aggregates from those events, and Reactors trigger side effects or create further events. Kickstarter implemented a minimal framework for d.rip using regular Active Record aggregates, a regular SQL database, synchronous atomic updates, and roughly 200 lines of home-made code. The approach supports replaying history to reconstruct past state, repair calculator bugs, backfill new aggregate fields, optimize read models, and forward events to analytics platforms, while presenting naming and mutation-to-event decomposition as important challenges.


### [MJIT: A Method Based Just-in-time Compiler for Ruby](https://yomu.fyi/post/mjit-a-method-based-just-in-time-compiler-for-ruby.md)
- Company: [Heroku](https://yomu.fyi/company/heroku.md)
- Author: Jonan Scheffler
- Published: Apr 12, 2018
- Type: Explainer

Ruby programs are compiled into YARV instructions and executed by a stack-oriented virtual machine, while register-oriented representations can expose more opportunities for compiler optimization. The post explains how YARV represents a simple expression, contrasts stack and register architectures, and introduces Vladimir Makarov’s Ruby RTL, an intermediate representation distinct from GCC’s RTL. MJIT uses RTL generated by MRI to make speculative, operand-specific optimizations, then compiles methods to machine code rather than relying only on interpretation. The interview discusses deoptimization: when assumptions change, MJIT generates a new machine-code version and can switch between JITed execution and RTL interpretation. The participants say this overhead should be tiny and should not degrade performance overall, while acknowledging edge cases where MJIT is not faster and that benchmarking JITs is difficult.


### [A Pods Architecture To Allow Shopify To Scale](https://yomu.fyi/post/a-pods-architecture-to-allow-shopify-to-scale.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: Xavier Denis
- Published: Mar 2, 2018
- Type: Explainer

Shopify outgrew its ability to scale by buying larger database servers, so it introduced database sharding in 2015. Sharding improved performance and horizontal scalability but made actions dependent on every shard, increasing the risk of platform-wide unavailability. In 2016, Shopify reorganized its runtime around pods: each pod groups shops on isolated datastores, while shared workers, application servers, and load balancers are restricted to one pod at a time. Sorting Hat assigns each request to a pod and location, while every web request and delayed job is kept within one pod; paired data centers and Pod Mover support recovery without dropping requests or jobs. The design makes a pod the unit for scalability, resiliency, and disaster recovery, and simplifies evacuating an entire data center.


### [GrabShare at the Intelligent Transportation Engineering Conference](https://yomu.fyi/post/grabshare-at-the-intelligent-transportation-engineering-conference.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Dominic Widdows
- Published: Dec 13, 2017
- Type: Announcement

Grab presented a technical paper on the construction of its real-time ridesharing service, GrabShare, at the Intelligent Transportation Engineering Conference in Singapore. The platform pairs passengers heading along similar routes with drivers immediately while handling network drops, volatile supply and demand, and heavy traffic conditions in Southeast Asian cities. To deliver accurate pairings, the scheduling system generates and filters through hundreds of travel time estimates for each candidate match before finalizing an itinerary. Operational teams on the ground evaluate complaints about poor matches, enabling engineers to refine the online matching systems. Over the course of one month, the service cut more than 4.5 million kilometers of driving distance and brought in over 100,000 new users within two weeks.


### [Namespacing Actions for Redux](https://yomu.fyi/post/namespacing-actions-for-redux.md)
- Company: [Kickstarter](https://yomu.fyi/company/kickstarter.md)
- Author: Sarah Groff Hennigh-Palermo
- Published: Nov 20, 2017
- Type: Explainer

Kickstarter’s front-end team encountered a Redux action-namespacing problem when multiple instances of the same functional React component responded to actions intended for only one instance. They rejected local state because the application used functional components and redux-loop, which routes state updates and side effects through reducers, and they sought programmatic namespacing compatible with combineReducers and a low-surface-area component API. The team compared nested reducers, higher-order reducers, and a closure-based module pattern; the latter met the stated constraints with namespaced actions, wrapper components, reducers, and a namespace returned by a factory that defaults to a UUID. Namespaces must be preserved when reducers trigger further actions, including redux-loop results, and complex composition can lose the required binding. The approach has worked for most reusable and compound components, while possible performance costs from many reducers remain a future concern.


### [FRP at Kickstarter and Kotlin](https://yomu.fyi/post/frp-at-kickstarter-and-kotlin.md)
- Company: [Kickstarter](https://yomu.fyi/company/kickstarter.md)
- Author: Julie
- Published: Nov 6, 2017
- Type: Tutorial

Kickstarter’s open-source iOS and Android apps use Functional Reactive Programming (FRP) patterns to keep views simple and move mutation and complex logic into thoroughly tested ViewModels. The post describes how this discipline helped a developer ramp up quickly on the Android codebase after working on Kickstarter’s backend and payments teams. It also introduces the Android team’s interest in Kotlin, a JetBrains language that runs on the JVM, while noting that dependency concerns had prevented conversion of core functionality. For a 2017 Functional Swift Conference talk in Berlin, the author used Kotlin to add a simple Android screen following the team’s established FRP patterns, presenting the implementation as live coding and an introduction to FRP and Kotlin.


[Newer posts](https://yomu.fyi/topic/architecture/page/40.md) · [Older posts](https://yomu.fyi/topic/architecture/page/42.md)