# Architecture > 335 posts about Architecture, summarised, each linking to the original. ## Articles ### [Plumbing At Scale](https://yomu.fyi/post/plumbing-at-scale.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Karan Kamath - Published: Jan 6, 2020 Grab's backend services process terabytes of data ingress per hour, generating recurring needs for stream transformations, joins, and time-windowed aggregations across diverse workloads. To support these asynchronous processing patterns across their Go ecosystem, the Coban team developed a managed, NoOps event sourcing and stream processing platform. The architecture packages stateless processing pipelines as Kubernetes deployments on AWS, polling Kafka event logs and using ScyllaDB as a shared metastore for stateful needs like deduplication and windowing. Stream processing pods combine ingestion triggers, a worker pool runtime, and user-provided domain logic plugins with customizable failure handling. This infrastructure scales to handle over 300 billion events weekly while maintaining workload isolation and elastic autoscaling. ### [How We Implemented Domain-Driven Development in Golang](https://yomu.fyi/post/how-we-implemented-domain-driven-development-in-golang.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Kapil Chaurasia - Published: Nov 21, 2019 Building GrabPlatform's partner integration self-service portal initially resulted in an unstructured codebase where individual files exceeded 500 lines and lacked proper segregation. Modifying existing functions carried high risks of breaking functionality across imported source collections. To resolve this, the team restructured the Go application using Domain-Driven Design principles in coordination with product domain experts. They mapped business rules into bounded contexts, identified entities and aggregate roots, introduced repository interfaces, and utilized domain events for cross-context communication. The refactoring distributed core functionality evenly, simplified onboarding, and aligned technical terminology with business concepts. ### [Griffin, an Anti-fraud Risk Rule Engine Making Billions of Predictions Daily](https://yomu.fyi/post/griffin-an-anti-fraud-risk-rule-engine-making-billions-of-predictions.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Muqi Li - Published: Oct 28, 2019 Grab's Trust/Identity/Safety team built Griffin, an in-house anti-fraud risk rule engine designed to process billions of daily predictions across multiple business verticals. Initially, Grab managed fraud rules directly within backend service code, but escalating rule complexity, tight rule interdependencies, and translation gaps between data scientists and developers caused deployment delays and misfiring errors. To overcome the limitations and steep learning curves of third-party engines like Drools, the team separated the workflow into data orchestration and rule-based prediction. Griffin enables analysts and data scientists to author Python-based rules directly via a web portal and reload updated logic into memory without manual developer intervention. By eliminating I/O during rule evaluation and leveraging Gunicorn multi-processing, the engine handles over 100,000 queries per second at peak on six EC2 instances with single-prediction latencies under six milliseconds. ### [Using Grab’s Trust Counter Service to Detect Fraud Successfully](https://yomu.fyi/post/using-grab-s-trust-counter-service-to-detect-fraud-successfully.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Chao Wang - Published: Oct 21, 2019 Grab's Trust Platform team built the Counter service to detect fraud across business verticals like transportation, food, and payments. The platform replaces manual, multi-week engineering cycles with a self-service UI where data analysts can define and experiment with counters independently. Operating on an asynchronous ingestion and synchronous transaction model, the architecture evaluates incoming stream data, enriches it via internal services, and persists aggregated signals to ScyllaDB through Grab-Stats. A multi-bucket strategy partitions queries into fifteen-minute, hourly, and daily granularities to maintain low-latency query aggregations across wide time ranges under strict SLAs. ### [Being a Principal Engineer at Grab](https://yomu.fyi/post/being-a-principal-engineer-at-grab.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Roman Atachiants - Published: Sep 25, 2019 Grab's rapid growth resulted in roughly 350 microservices powering its superapp, creating the need for defined individual contributor career milestones. At Grab, a principal engineer oversees the architecture of an entire Tech Family, a sub-organisation containing over 50 engineers and 20 or more microservices. Responsibilities include translating broad, ambiguous problems into concrete projects, managing technical debt, and aligning multiple engineering teams across global R&D centres. The role demands continuous technical leadership through RFC design reviews, cross-functional communication, mentorship, and self-directed prioritization alongside engineering leadership. Ultimately, principal engineers amplify engineering quality and operational stability without directly managing people. ### [No More Forgetting to Input ERP Charges - Hello Automated ERP!](https://yomu.fyi/post/no-more-forgetting-to-input-erp-charges-hello-automated-erp.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Garvee Garg - Published: Jul 31, 2019 Grab launched an automated Electronic Road Pricing (ERP) fare calculation feature in Singapore to eliminate the need for driver-partners to manually track gantries and enter toll charges. Because Singapore gantries frequently adjust fares based on time and road conditions, manual entry often caused driver errors and revenue loss. Grab solved this by mapping precise geographical coordinates for every toll gate using satellite imagery and open data, matching frequent driver GPS pings against road layers and gantry locations. The engineering and operations teams also built an internal ERP Workflow tool to map ride trajectories and resolve driver dispute feedback within an average of one day. Following its rollout in Singapore, Grab began testing and planning regional expansion to Indonesia, Thailand, Malaysia, and the Philippines. ### [How We Built a Logging Stack at Grab](https://yomu.fyi/post/how-we-built-a-logging-stack-at-grab.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Daniel Kasen - Published: Jul 31, 2019 Grab needed a scalable logging platform to replace slow, fragmented systems that hindered debugging across their growing service fleet. Generating 25TB of daily logs, the team built a horizontally scalable Elasticsearch cluster configured via Ansible and monitored with Datadog. Although the initial proof of concept assigned all node roles (ingest, coordinator, master, and data) to every machine, operating at scale introduced major challenges with JVM heap exhaustion and cluster stability. The team resolved memory pressure and performance bottlenecks by tuning circuit breakers, lowering field data cache limits, adjusting shard allocations based on segment memory, and disabling translog compression during shard transfers. ### [Making Grab’s Everyday App Super](https://yomu.fyi/post/making-grab-s-everyday-app-super.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Justin Bolilia - Published: Jul 3, 2019 Grab manages an expanding superapp ecosystem comprising ride-hailing, food delivery, payments, and partner content surfaced through the Grab Feed. As content volume grows, the platform risks overwhelming users with irrelevant information. To address this, Grab built a recommendation engine that ranks cards using signals across user profiles, content metadata, and contextual factors such as time and location. The system employs multiple recommendation strategies—including popularity metrics, user favorites, collaborative filtering, habitual patterns, and cross-platform deep embeddings—which are selected or aggregated. Recommendation quality is evaluated via offline metrics like Recall@K and NDCG alongside online engagement experiments. ### [Catwalk: Serving Machine Learning Models at Scale](https://yomu.fyi/post/catwalk-serving-machine-learning-models-at-scale.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Nutdanai Phansooksai - Published: Jul 2, 2019 As machine learning adoption expanded at Grab, individual teams created fragmented model serving solutions that duplicated engineering effort and required data scientists to handle underlying infrastructure. To resolve these inefficiencies, Grab developed Catwalk, a self-service machine learning model serving platform. The system runs TensorFlow Serving containers across a managed Kubernetes cluster integrated with Grab's observability stack. Data scientists deploy or update models simply by saving files using the tf.saved\_model API to dedicated Amazon S3 buckets, while Kubernetes automates orchestration, ingress routing, and pod autoscaling. Catwalk abstracts server management away from data scientists, shortens deployment timelines, and provides high availability during model version rollouts. ### [React Native in GrabPay](https://yomu.fyi/post/react-native-in-grabpay.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Sushant Tiwari - Published: May 30, 2019 Following the release of the GrabPay Merchant App, Grab adopted React Native inside the Grab Passenger app to maintain a single cross-platform codebase across iOS and Android. Integrating the framework into native applications required establishing bridge communication guidelines and incorporating React Native modules into the Grablet architecture. To streamline network communication, API calls were migrated from axios to native bridges returning promises, which eliminated the need to pass access tokens into JavaScript. The team also established a shared internal library of approximately 20 UI components alongside Redux for state management and react-navigation for routing. Modules like BillPay and Transaction History successfully launched across Southeast Asia while maintaining the performance and feel of native software. ### [Connecting the Invisibles to Design Seamless Experiences](https://yomu.fyi/post/connecting-the-invisibles-to-design-seamless-experiences.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Stephanie Lukito - Published: May 28, 2019 Service design operates as connective tissue within complex product ecosystems by bridging digital touchpoints, physical operations, and backstage technical workflows. At Grab, placing an order on GrabFood requires coordinating driver allocation, customer support paths, and long-term data storage rather than simply transmitting information to merchants. Focusing exclusively on singular features risks breaking broader network dependencies when modifications ripple into other operational systems. Grab addresses these interdependencies through participatory design processes and visual mapping across cross-functional teams. This holistic framework evaluates whether systemic issues, such as inaccurate restaurant operating hours, are best resolved through in-app feature changes or operational adjustments. ### [Designing Resilient Systems Beyond Retries (Part 3): Architecture Patterns and Chaos Engineering](https://yomu.fyi/post/designing-resilient-systems-beyond-retries-part-3-architecture-pattern.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Michael Cartmell - Published: Mar 27, 2019 Building resilient systems requires architectural safeguards and proactive testing beyond basic retries and circuit breakers. Architectural patterns such as idempotency keys enable safe retries without creating inconsistent state during failures. Asynchronous responses and deferrable work isolate services from downstream dependency latency and errors, though they can conflict with the fail-fast principle. To validate system behavior under stress, chaos engineering introduces intentional failures in production to test hypotheses against a defined steady state. Selectively adopting complementary patterns reduces failure points while avoiding unnecessary architectural complexity. ### [Designing Resilient Systems Beyond Retries (Part 2): Bulkheading, Load Balancing, and Fallbacks](https://yomu.fyi/post/designing-resilient-systems-beyond-retries-part-2-bulkheading-load-bal.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Michael Cartmell - Published: Mar 25, 2019 Software systems require mechanisms beyond retries to maintain resilience during downstream outages and high traffic. Bulkheading isolates failures across infrastructure, processes, thread pools, and connection limits, preventing a single failing component from degrading an entire system. Load balancing distributes traffic across backend pools via proxies, client-side libraries, lookaside services, or sidecars, often pairing with health checks to eliminate single points of failure. When operations fail unrecoverably, fallback strategies like silent failures, local defaults, stale cache reads, and dedicated backup services enable graceful degradation. Organizations like Grab implement these approaches using internal client-side load balancers backed by etcd, cache fallbacks in microservice frameworks, and redundant core backup services. ### [Designing Resilient Systems Beyond Retries (Part 1): Rate-Limiting](https://yomu.fyi/post/designing-resilient-systems-beyond-retries-part-1-rate-limiting.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Michael Cartmell - Published: Mar 20, 2019 Distributed systems that rely exclusively on retries and circuit breakers face severe failure risks, including retry storms and reliance on client-side configuration accuracy. Implementing server-side rate limiting serves as a critical defensive layer to safeguard services across evolving architectures. Throttling thresholds can be layered across per-client, per-endpoint, and server-wide granularities using algorithms such as leaky bucket or sliding windows. While local instance-level limits fail when downstream bottlenecks like databases saturate under horizontal scaling, global rate limiting coordinates traffic enforcement across entire service pools. Centralized rate limiters require asynchronous communication and fallback mechanisms to avoid becoming single points of failure or adding request path latency. ### [Context Deadlines and How to Set Them](https://yomu.fyi/post/context-deadlines-and-how-to-set-them.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Michael Cartmell - Published: Mar 11, 2019 Microservice architectures operating under heavy network traffic require robust timeout handling to prevent slow or failing dependencies from causing cascading failures across services. Naive, static timeout configurations across a call chain often cause downstream components to waste compute effort on requests that upstream callers have already abandoned. To establish predictable timeout thresholds, engineers can align limits with service-level latency percentiles, pairing P99 limits with median latency retry allowances. Go's context package improves upon static network timeouts by propagating request-scoped deadlines and cancellation signals across service boundaries. This distributed context ensures downstream servers recognize remaining time budgets and terminate unneeded processing immediately when parent deadlines expire or callers manually cancel requests. ### [Structured Logging: The Best Friend You’ll Want When Things Go Wrong](https://yomu.fyi/post/structured-logging-the-best-friend-you-ll-want-when-things-go-wrong.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Aditya Praharaj - Published: Mar 5, 2019 Grab redesigned its backend logging approach to address mounting vendor costs, query language limitations, and debugging difficulties across a growing microservices ecosystem. Most services previously emitted syslog-style key-value logs almost entirely at the INFO level, which made volume reduction difficult and lacked causal ordering and automated correlation. The engineering team migrated to a self-managed Elastic stack backend and built a structured logging library in Go from the ground up. This framework introduces dynamic log-level adjustment at runtime, automatic trace-based log correlation via Grab-Kit, and the Common Grab Log Schema to enforce consistent JSON formatting without Elasticsearch indexing conflicts. ### [A Lean and Scalable Data Pipeline to Capture Large Scale Events and Support Experimentation Platform](https://yomu.fyi/post/a-lean-and-scalable-data-pipeline-to-capture-large-scale-events-and-su.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Oscar Cassetti - Published: Jan 16, 2019 Controlled online experimentation across diverse product verticals requires tracking interactions across systems to prevent local optimizations from causing global degradation. Grab built a batch data pipeline to capture, ingest, and process petabytes of event data to support its experimentation platform and analytics stakeholders. The architecture loads ingested event data from Amazon S3, transforms and sorts it, and writes partitioned output back to S3 with metadata registered in Apache Hive. Using Apache Spark on AWS Elastic MapReduce with Apache Airflow for orchestration, the system handles roughly 400,000 incoming events per second. The data is partitioned by event type and ingestion time and stored in Apache ORC format to streamline query workloads and reduce retrieval overhead. ### [Designing Resilient Systems: Circuit Breakers or Retries? (Part 2)](https://yomu.fyi/post/designing-resilient-systems-circuit-breakers-or-retries-part-2.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Corey Scott - Published: Jan 8, 2019 Retries enable software systems to recover from transient upstream failures by automatically repeating unsuccessful requests. While retrying increases the chance of request completion across multi-host setups, it consumes additional CPU and time without inherently tracking host health. Applications must selectively retry errors with a likelihood of success, such as 500 and 503 status codes, while avoiding client-side failures like 400 or 401. To manage distributed systems safely, retries require idempotent operations or cryptographic nonces, along with backoff and jitter to prevent request stampedes. Tuning retry counts, timeouts, and delays is critical to cap the worst-case consumer response time. ### [Querying Big Data in Real-time with Presto & Grab's TalariaDB](https://yomu.fyi/post/querying-big-data-in-real-time-with-presto-grab-s-talariadb.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Roman Atachiants - Published: Jan 2, 2019 Grab developed TalariaDB to support real-time SQL querying over high-velocity event streams while maintaining predictable sub-second latencies and low infrastructure costs. The distributed time-series store retains only the most recent hour of data and integrates directly with Presto via its PrestoThriftService interface. Internally, TalariaDB uses the Go-based Badger key-value store to maintain an in-memory key index of metric names and timestamps while mapping columnar event payloads directly to disk. Ingestion occurs by processing pre-partitioned event batches written to Amazon S3 via SQS notifications. By combining a zero-copy decoder with parallel split evaluation across gossiping cluster nodes, the architecture scales horizontally while serving millions of events per second. ### [Designing Resilient Systems: Circuit Breakers or Retries? (Part 1)](https://yomu.fyi/post/designing-resilient-systems-circuit-breakers-or-retries-part-1.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Corey Scott - Published: Dec 21, 2018 Distributed architectures frequently encounter upstream failures triggered by networking issues, system overloads, resource starvation, and invalid deployments. Implementing software circuit breakers interposes a monitoring mechanism between components to halt requests when failure thresholds are met, giving struggling upstream dependencies time to recover. Circuit breakers save CPU, memory, and network resources by failing fast or routing execution through defined fallbacks such as cached data, alternate services, or approximation algorithms. Grab utilizes Hystrix-Go to manage upstream interactions and configure key thresholds for concurrency, timeouts, and error ratios. This approach protects downstream consumers from cascading latency while insulating upstream resources from excess traffic. [Newer posts](https://yomu.fyi/topic/architecture/page/15.md) · [Older posts](https://yomu.fyi/topic/architecture/page/17.md)