# Architecture > 329 posts about Architecture, summarised, each linking to the original. ## Articles ### [Supabase Launches NFT Marketplace](https://yomu.fyi/post/supabase-launches-nft-marketplace.md) - Company: [Supabase](https://yomu.fyi/company/supabase.md) - Author: Ant Wilson - Published: Apr 1, 2021 Non-fungible tokens frequently suffer from the copy-paste problem, where public URLs allow unauthorized users to download underlying media directly from web hosts or IPFS. To address this issue, BuyMeth proposes combining public blurhash thumbnails with full image files encrypted under the active owner's cryptographic key and hosted on IPFS. Completed sales trigger Metamask to re-encrypt the file with the buyer's public key, followed by a one-week escrow challenge window where automated verification matches decrypted image hashes against public thumbnails. Ongoing royalty distributions to previous owners provide a financial mechanism to disincentivize leaking unencrypted original image files. The announced platform serves as an April Fools joke rather than an active Supabase product release. ### [Storage is now available in Supabase](https://yomu.fyi/post/storage-is-now-available-in-supabase.md) - Company: [Supabase](https://yomu.fyi/company/supabase.md) - Author: Inian Parameshwaran - Published: Mar 30, 2021 Supabase launched Storage, adding a scalable object store to its existing Postgres, authentication, and API services. Existing open-source storage servers like Ceph, Swift, Minio, and Zenko were evaluated but rejected due to auth incompatibilities and external dependencies like etcd, MongoDB, and Kafka. Consequently, the team implemented a custom Storage API server built with Fastify and TypeScript behind the Kong gateway. Object metadata and access control reside directly in Postgres, leveraging Row Level Security policies written in SQL rather than a proprietary domain-specific language. Objects stream directly to managed backends like AWS S3 using Node streams with minimal in-memory buffering, accompanied by a default one-hour Cache-Control header. ### [Customer Support Workforce Routing](https://yomu.fyi/post/customer-support-workforce-routing.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Suman Anand - Published: Feb 5, 2021 Grab replaced its third-party customer support routing software with an in-house workforce routing system for Livechat to gain better priority controls, bespoke configurations, and deeper analytics. The platform separates requests into distinct priority and business queues, using parallel workers that spend varied time slices dequeuing higher-priority issues like safety concerns. To prevent request starvation, workers operate out of sync across queue priority levels while dynamic queue limits cap incoming volume based on agent availability and performance. The system routes requests through an intermediate Agent Group layer, calculating eligibility scores from proficiency and concurrency metrics while managing per-agent locks to prevent over-allocation. ### [Serving Driver-partners Data at Scale Using Mirror Cache](https://yomu.fyi/post/serving-driver-partners-data-at-scale-using-mirror-cache.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Indrajit Sarkar - Published: Jan 26, 2021 Grab's Drivers Data service handles up to 10,000 requests per second during peak hours to supply driver information across backend microservices. The original setup used MySQL with Redis and standalone in-memory local caches, but yielded a low 25% local cache hit rate due to traffic patterns characterized by high burst frequency for individual drivers alongside redundant database calls across nodes. To solve this, the team developed Mirror Cache, an in-memory caching system that pairs Dgraph's Ristretto library with an asynchronous gRPC replication layer to mirror updates across cluster nodes. The replicator batches updates within the same AWS availability zone and forwards data to single nodes across zones to minimize transfer overhead. Production deployment increased the in-memory cache hit rate to approximately 75% and reduced direct MySQL queries by 5%. ### [The GrabMart Journey](https://yomu.fyi/post/the-grabmart-journey.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Clarisse Peralta - Published: Jan 18, 2021 During the COVID-19 pandemic, Grab accelerated the rollout of its GrabMart on-demand grocery delivery service to meet surging regional demand. Early pilot testing within the existing GrabFood interface revealed that grocery shopping fundamentally diverges from meal ordering, as users search at an item level rather than browsing merchants. To support grocery workflows, the team restructured the user experience with item and store categories, one-click add-to-cart functionality, and scheduled delivery options. Engineering teams simultaneously updated core infrastructure across order management, driver allocation systems, and merchant Partner APIs to handle specialized mart bookings. Rollouts were validated across two main phases using the internal Grab Early Access program to catch issues before regional release. ### [Trident - Real-time Event Processing at Scale](https://yomu.fyi/post/trident-real-time-event-processing-at-scale.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Jie Zhang - Published: Jan 13, 2021 Trident serves as Grab's internal real-time event-processing and workflow automation engine, driving user campaigns, rewards, and notifications across multiple business lines. To handle peak loads exceeding 2,000 events per second without duplicate execution, the system consumes decoupled Kafka streams and enforces exactly-once semantics using Redis and MySQL deduplication checks. Processing efficiency relies on server autoscaling aligned with Kafka partition counts, combined with dynamic goroutine allocation per consumer. To minimize rule evaluation overhead, Trident indexes active campaigns into an in-memory hash map by event type, cutting processing time by at least 90%. Furthermore, condition evaluation is optimized through lazy loading and a weighted sorting algorithm that checks low-cost in-memory data prior to executing expensive database queries or external service calls. ### [Pharos - Searching Nearby Drivers on Road Network at Scale](https://yomu.fyi/post/pharos-searching-nearby-drivers-on-road-network-at-scale.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Hao Wu - Published: Dec 22, 2020 Ride-hailing allocation requires identifying nearby drivers based on actual routing distance and estimated time of arrival rather than straight-line haversine distance. Grab developed Pharos, a distributed in-memory spatial microservice that evaluates road networks to match fast-moving drivers with pick-up requests. The system partitions OpenStreetMap road graphs by city and vehicle vertical while indexing driver positions and edge-based nodes using Adaptive Radix Trees. To locate candidates, Pharos projects driver coordinates onto road segments as phantom nodes and runs Incremental Network Expansion to compute isochrone reaches. Read-write concurrency is managed by switching tree root snapshots, allowing concurrent reads while updates produce new roots for subsequent lookups. ### [Democratising Fare Storage at Scale Using Event Sourcing](https://yomu.fyi/post/democratising-fare-storage-at-scale-using-event-sourcing.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Sourabh Suman - Published: Nov 23, 2020 Grab's legacy system stored booking and fare details in a single relational table, creating a bloated booking entity that tracked only the latest fare state and hindered rapid feature iteration. To resolve scalability, stability, and debugging challenges across millions of daily bookings, the team developed Fare Storage using the Event Sourcing pattern. The new architecture persists all fare modification events chronologically in DynamoDB, backed by a cache for eventually consistent reads and message streaming for downstream processing. The platform employs optimistic locking with versioning to manage concurrent updates, enforces idempotency through client-generated transaction UUIDs, and delegates metadata serialization to an SDK to prevent storage API changes. ### [Securing and Managing Multi-cloud Presto Clusters with Grab’s DataGateway](https://yomu.fyi/post/securing-and-managing-multi-cloud-presto-clusters-with-grab-s-datagate.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Vinnson Lee - Published: Aug 24, 2020 Grab's data engineering team needed to manage data access across growing Presto workloads spanning hundreds of users, thousands of tables, and multiple clouds without exposing infrastructure churn to clients. Out-of-the-box Presto lacked fine-grained schema- and table-level access controls, unified endpoint routing, and seamless cluster switching. To solve this, the team built DataGateway, a smart HTTP proxy and abstraction layer deployed as microservices on Kubernetes. The platform parses incoming SQL statements, verifies user access control lists against target schemas and tables, and dynamically routes queries to appropriate AWS or Azure Presto clusters without requiring endpoint updates. This architecture enabled zero-disruption cluster migrations across 40 clusters and unified access management across a multi-cloud data lake environment. ### [Supabase Auth](https://yomu.fyi/post/supabase-auth.md) - Company: [Supabase](https://yomu.fyi/company/supabase.md) - Author: Paul Copplestone - Published: Aug 5, 2020 Following an early community launch on Hacker News, the open-source Firebase alternative Supabase faced heavy developer demand for a dedicated authentication and authorization solution. In response, the team evaluated existing open-source tools such as KeyCloak and Kratos before landing on an architecture combining a forked Netlify GoTrue server with PostgreSQL and PostgREST. The resulting Supabase Auth system provides client methods like signUp and signIn alongside authorization powered by PostgreSQL Row Level Security and policies. By leveraging SQL policies to enforce access rules directly in the database, developers avoid repetitive middleware filtering while maintaining granular access controls. Future roadmap plans include adding third-party OAuth providers, custom email templates, and simplified policy management interfaces for non-technical users. ### [The Journey of Deploying Apache Airflow at Grab](https://yomu.fyi/post/the-journey-of-deploying-apache-airflow-at-grab.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Chandulal Kavar - Published: Jul 14, 2020 Engineering and data teams across Grab originally operated independent Apache Airflow instances, causing duplicate maintenance overhead and frequent job failures around scaling, logging, and dependency management. To resolve this, a dedicated team developed a centralized orchestration platform that runs isolated, containerized Airflow instances per team on Amazon EKS. The platform categorizes deployments into three size tiers and provisions dedicated Redis brokers, RDS metadata stores, and Vault secret sidecars using Terraform and custom Helm charts. Teams customize container images using shared GitLab CI/CD templates, while worker scaling is handled via Kubernetes Horizontal Pod Autoscalers. Today, the platform runs roughly 20 Airflow instances executing between 1,000 and 60,000 daily jobs per instance. ### [The Reformer - Pushing the limits of language modeling](https://yomu.fyi/post/the-reformer-pushing-the-limits-of-language-modeling.md) - Company: huggingface.co - Author: Patrick von Platen - Published: Jul 3, 2020 Standard transformer models hit memory bottlenecks on long sequence modeling tasks due to the quadratic asymptotic memory complexity of global self-attention and oversized positional embedding matrices. The Reformer architecture overcomes these constraints to train sequences of up to half a million tokens using under 8GB of RAM. It re-engineers transformer operations using local and Locality Sensitive Hashing self-attention, chunked feed forward layers, reversible residual layers, and axial positional encodings. In empirical benchmarks using google/reformer-crime-and-punishment, axial positional encodings reduce the model parameter count from over 136 million to approximately 2.58 million by factorizing the positional dimensions. This architectural change cuts inference memory consumption from 959 MB down to 447 MB for evaluated benchmark workloads. ### [How We Built Our In-house Chat Platform for the Web](https://yomu.fyi/post/how-we-built-our-in-house-chat-platform-for-the-web.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Vasu Krishnamoorthy - Published: Jun 29, 2020 Grab extended its in-house chat platform to the web to support its internal Customer Support portal. Because the existing TCP gateway only supports unicast connections with one active connection per user, opening multiple browser tabs would repeatedly disconnect previous tabs. Rather than undertaking a complex migration to multicast connections on the server, the team adopted a hybrid client-side strategy using SharedWorker and BroadcastChannel APIs. The implementation uses a SharedWorker to maintain a single WebSocket connection per domain while a BroadcastChannel syncs events across all open tabs. A custom wrapper over the worker manages version transitions during deployments to avoid race conditions across tabs. ### [How We Prevented App Performance Degradation from Sudden Ride Demand Spikes](https://yomu.fyi/post/how-we-prevented-app-performance-degradation-from-sudden-ride-demand-s.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Corey Scott - Published: Jan 8, 2020 Grab experienced severe system strain when sudden localized spikes in ride demand, triggered by events like heavy rain or concert dismissals, coincided with driver shortages. These localized bursts overloaded the platform and degraded the experience for users outside the affected areas. To mitigate this, engineers created the Spampede filter, a circuit-breaker mechanism placed at the start of the booking pipeline. The filter converts pickup locations into Geohash Integer buckets and partitions time using Unix timestamps, tracking unfulfilled requests in Redis with atomic increments and time-to-live expirations. When unallocated requests exceed configured thresholds within a specific bucket, the system immediately short-circuits new incoming bookings to protect overall platform stability. ### [Plumbing At Scale](https://yomu.fyi/post/plumbing-at-scale.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Karan Kamath - Published: Jan 6, 2020 Grab's backend services process terabytes of data ingress per hour, generating recurring needs for stream transformations, joins, and time-windowed aggregations across diverse workloads. To support these asynchronous processing patterns across their Go ecosystem, the Coban team developed a managed, NoOps event sourcing and stream processing platform. The architecture packages stateless processing pipelines as Kubernetes deployments on AWS, polling Kafka event logs and using ScyllaDB as a shared metastore for stateful needs like deduplication and windowing. Stream processing pods combine ingestion triggers, a worker pool runtime, and user-provided domain logic plugins with customizable failure handling. This infrastructure scales to handle over 300 billion events weekly while maintaining workload isolation and elastic autoscaling. ### [How We Implemented Domain-Driven Development in Golang](https://yomu.fyi/post/how-we-implemented-domain-driven-development-in-golang.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Kapil Chaurasia - Published: Nov 21, 2019 Building GrabPlatform's partner integration self-service portal initially resulted in an unstructured codebase where individual files exceeded 500 lines and lacked proper segregation. Modifying existing functions carried high risks of breaking functionality across imported source collections. To resolve this, the team restructured the Go application using Domain-Driven Design principles in coordination with product domain experts. They mapped business rules into bounded contexts, identified entities and aggregate roots, introduced repository interfaces, and utilized domain events for cross-context communication. The refactoring distributed core functionality evenly, simplified onboarding, and aligned technical terminology with business concepts. ### [Griffin, an Anti-fraud Risk Rule Engine Making Billions of Predictions Daily](https://yomu.fyi/post/griffin-an-anti-fraud-risk-rule-engine-making-billions-of-predictions.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Muqi Li - Published: Oct 28, 2019 Grab's Trust/Identity/Safety team built Griffin, an in-house anti-fraud risk rule engine designed to process billions of daily predictions across multiple business verticals. Initially, Grab managed fraud rules directly within backend service code, but escalating rule complexity, tight rule interdependencies, and translation gaps between data scientists and developers caused deployment delays and misfiring errors. To overcome the limitations and steep learning curves of third-party engines like Drools, the team separated the workflow into data orchestration and rule-based prediction. Griffin enables analysts and data scientists to author Python-based rules directly via a web portal and reload updated logic into memory without manual developer intervention. By eliminating I/O during rule evaluation and leveraging Gunicorn multi-processing, the engine handles over 100,000 queries per second at peak on six EC2 instances with single-prediction latencies under six milliseconds. ### [Using Grab’s Trust Counter Service to Detect Fraud Successfully](https://yomu.fyi/post/using-grab-s-trust-counter-service-to-detect-fraud-successfully.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Chao Wang - Published: Oct 21, 2019 Grab's Trust Platform team built the Counter service to detect fraud across business verticals like transportation, food, and payments. The platform replaces manual, multi-week engineering cycles with a self-service UI where data analysts can define and experiment with counters independently. Operating on an asynchronous ingestion and synchronous transaction model, the architecture evaluates incoming stream data, enriches it via internal services, and persists aggregated signals to ScyllaDB through Grab-Stats. A multi-bucket strategy partitions queries into fifteen-minute, hourly, and daily granularities to maintain low-latency query aggregations across wide time ranges under strict SLAs. ### [Being a Principal Engineer at Grab](https://yomu.fyi/post/being-a-principal-engineer-at-grab.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Roman Atachiants - Published: Sep 25, 2019 Grab's rapid growth resulted in roughly 350 microservices powering its superapp, creating the need for defined individual contributor career milestones. At Grab, a principal engineer oversees the architecture of an entire Tech Family, a sub-organisation containing over 50 engineers and 20 or more microservices. Responsibilities include translating broad, ambiguous problems into concrete projects, managing technical debt, and aligning multiple engineering teams across global R&D centres. The role demands continuous technical leadership through RFC design reviews, cross-functional communication, mentorship, and self-directed prioritization alongside engineering leadership. Ultimately, principal engineers amplify engineering quality and operational stability without directly managing people. ### [No More Forgetting to Input ERP Charges - Hello Automated ERP!](https://yomu.fyi/post/no-more-forgetting-to-input-erp-charges-hello-automated-erp.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Garvee Garg - Published: Jul 31, 2019 Grab launched an automated Electronic Road Pricing (ERP) fare calculation feature in Singapore to eliminate the need for driver-partners to manually track gantries and enter toll charges. Because Singapore gantries frequently adjust fares based on time and road conditions, manual entry often caused driver errors and revenue loss. Grab solved this by mapping precise geographical coordinates for every toll gate using satellite imagery and open data, matching frequent driver GPS pings against road layers and gantry locations. The engineering and operations teams also built an internal ERP Workflow tool to map ride trajectories and resolve driver dispute feedback within an average of one day. Following its rollout in Singapore, Grab began testing and planning regional expansion to Indonesia, Thailand, Malaysia, and the Philippines. [Newer posts](https://yomu.fyi/topic/architecture/page/14.md) · [Older posts](https://yomu.fyi/topic/architecture/page/16.md)