# Architecture
> 320 posts about Architecture, summarised, each linking to the original.

## Articles

### [The complete stream processing journey on FlinkSQL](https://yomu.fyi/post/the-complete-stream-processing-journey-on-flinksql.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Calvin Tran
- Published: Jun 12, 2025

Grab previously relied on Apache Zeppelin notebooks for interactive stream processing exploration, but faced lagging Flink version upgrades, five-minute cluster cold starts, and poor integration with internal platforms. To address these limitations, the team migrated to a shared FlinkSQL gateway architecture structured into compute, integration, and query layers. The new setup uses a Hive Metastore catalog to expose Kafka topics as relational tables, while a custom control plane handles authentication and headless REST APIs over Flink's native interface. For production workflows, a configuration-based portal accepts SQL logic and automatically provisions and deploys Flink pipelines within ten minutes. This transition reduced ad-hoc query response times to under one minute and eliminated the need to maintain version adapter shims.


### [How We Decomposed Tinder’s Monolith](https://yomu.fyi/post/how-we-decomposed-tinder-s-monolith.md)
- Company: [Tinder](https://yomu.fyi/company/tinder.md)
- Author: Tinder
- Published: May 23, 2025

Tinder faced significant agility and build performance challenges caused by an iOS codebase monolith containing over 1,000 files and 150,000 lines of code. Manual extraction efforts risked creating massive, unmanageable pull requests that would require constant rebasing against the main branch. To systematically decompose the target into Swift sub-modules, the team mapped declarations and references via the Swift compiler into a directed graph. They iteratively extracted leaf nodes with an in-degree of zero across sequential phases and automated common code adjustments, including module dependencies, imports, access control levels, and dependency injection. The automated decomposition completed in under six months with zero P0 incidents, reducing monolith build times by 78% and disallowing future additions to the monolith target.


### [Effortless enterprise authentication at Grab: Dex in action](https://yomu.fyi/post/effortless-enterprise-authentication-at-grab-dex-in-action.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Kah Wei Lee
- Published: May 23, 2025

Grab needed a centralised system to simplify identity management, satisfy audit requirements, and standardise authentication across internal and external tools like Databricks and Datadog. The engineering team selected OpenID Connect as their standard protocol and adopted Dex, an open-source CNCF identity aggregator. Dex acts as an intermediary between applications and multiple identity providers to issue standardised OIDC tokens. To secure service-to-service communication, Grab implemented token exchange with trusted peer relationships rather than relying on privileged service accounts. Dex also provides a kill-switch mechanism that can route authentication traffic to an alternate provider during identity provider outages.


### [Streamlining RiskOps with the SOP agent framework](https://yomu.fyi/post/streamlining-riskops-with-the-sop-agent-framework.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Fujiao Liu
- Published: May 8, 2025

Manual Account Takeover (ATO) investigations in Risk Operations traditionally demand intensive cross-referencing across systems, manual SQL execution, and high-pressure decision-making prone to human error. To resolve these bottlenecks, an SOP-driven LLM agent framework models investigative workflows as natural-language tree structures with explicit function notations like @function\_name. Execution is coordinated between an SOP planner, which traverses the tree using a Depth-First Search strategy, and a Worker Agent that parses JSON-formatted steps to invoke database queries and APIs. Once all steps evaluate their decision criteria, the framework synthesizes the collected data into an actionable summary report. Implementing this architecture automated 87% of ATO cases and dropped average ticket handling time from 22 minutes to 3 minutes.


### [Introducing the SOP-driven LLM agent frameworks](https://yomu.fyi/post/introducing-the-sop-driven-llm-agent-frameworks.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Fujiao Liu
- Published: Apr 25, 2025

Standard operating procedure (SOP) driven Large Language Model agent frameworks address common generative AI challenges such as hallucinations, non-standard output formats, and branching navigation errors. Workflows are represented as hierarchical trees where nodes encapsulate actions or decision points that can be created using a visual editor and annotated with explicit external function calls. Execution relies on a tripartite architecture consisting of a Depth-First Search planner module with backtracking, an adaptive worker agent that limits API exposure and compresses context, and a multilingual user agent. Supporting tools include a Graph Retrieval-Augmented Generation pipeline, a plugin system integrating Python and SQL, and a state stack for pausing workflows during human intervention. In production deployments for fraud and account takeover investigations, the framework automated up to 87% of cases while cutting handling times substantially.


### [Evaluating performance impact of removing Redis-cache from a Scylla-backed service](https://yomu.fyi/post/evaluating-performance-impact-of-removing-redis-cache-from-a-scylla-ba.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Md Riyadh
- Published: Apr 11, 2025

Grab operates a high-throughput Rust read service that aggregates counter metrics from Scylla tables across minutely, hourly, and daily granularities. The service initially cached aggregated responses in Redis using keys rounded to 15-minute intervals alongside a five-minute TTL. Because incoming queries predominantly requested recent data, transitions between 15-minute windows caused simultaneous cache misses across active configurations, resulting in severe Scylla traffic spikes, latency surges, and timeouts. To resolve the load imbalance, engineers proposed removing the Redis cache entirely and relying directly on Scylla's native internal caching. The rollout was staged in production by deterministically disabling Redis caching for specific counter configurations using mathematical operations on configuration IDs.


### [Scaling Nextdoor’s Datastores: Part 3](https://yomu.fyi/post/scaling-nextdoor-s-datastores-part-3.md)
- Company: [Nextdoor](https://yomu.fyi/company/nextdoor.md)
- Author: Ronak Shah
- Published: Mar 19, 2025

Look-aside caching with object byte serialization can cause critical compatibility failures when application versions, dependencies, or database schemas change. Serialized cache entries bound to specific runtimes risk deserialization errors during deployments, triggering thundering herd queries against the underlying datastore. To resolve this, Nextdoor replaced runtime-bound serialization like Python pickle with MessagePack to serialize Django model objects. The team achieved forward compatibility by letting MessagePack ignore unrecognized new fields in older application code, while backward compatibility relies on mandatory default values for newly added model attributes. Nextdoor prepends a ten-byte header containing format metadata and version information before writing the serialized payload to cache stores.


### [Scaling Nextdoor’s Datastores: Part 2](https://yomu.fyi/post/scaling-nextdoor-s-datastores-part-2.md)
- Company: [Nextdoor](https://yomu.fyi/company/nextdoor.md)
- Author: Tushar Singla
- Published: Mar 19, 2025

Nextdoor encountered scaling issues after adding read replicas when product engineers were initially tasked with deciding whether to route queries to the primary or replica databases. As business logic grew and gained abstraction layers, engineers struggled to track read-after-write consistency constraints across the call stack. To avoid replication lag race conditions, engineers routinely wrapped logic in database transactions, unintentionally directing all queries to the primary node and eroding read replica benefits over several years. The Core-Services team resolved this by injecting custom tracking logic into their Django ORM layer to monitor table writes during web requests and automate routing. They further optimized the system using a timing-based approach that restored replica read eligibility after the p99.9 replication lag elapsed.


### [Grab AI Gateway: Connecting Grabbers to multiple GenAI providers](https://yomu.fyi/post/grab-ai-gateway-connecting-grabbers-to-multiple-genai-providers.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Bjorn Jee
- Published: Feb 19, 2025

Grab built the AI Gateway to centralize access, cost control, and security across external and open-source Generative AI providers such as OpenAI, Azure, AWS, and Google. Designed as a set of lightweight reverse proxies, the gateway manages authentication, rate limiting, and authorization while translating payloads into a unified OpenAI-compatible interface. The platform archives request metadata and calculated per-call costs into a central data lake for auditing and showback, dynamically routing traffic across shared reserved capacity and regions to mitigate quota throttling. Supporting over 300 internal use cases, the system integrates directly with internal development notebooks and deployment tools to power applications ranging from real-time audio safety analysis to automated content moderation.


### [Sharing Tinder’s latest contributions to the open source community](https://yomu.fyi/post/sharing-tinder-s-latest-contributions-to-the-open-source-community.md)
- Company: [Tinder](https://yomu.fyi/company/tinder.md)
- Author: Tinder
- Published: Jan 29, 2025

Tinder open-sourced several iOS development repositories, including Layout, Nodes Architecture Framework, and CombineUI, to share the engineering patterns supporting its iPhone application. As the app expanded across diverse device profiles and scale, engineering teams faced reliability, consistency, and memory challenges under their legacy architecture. In response, Tinder developed Nodes, a plugin-based architecture framework using compile-time dependency injection and lifecycle hooks that enforce complete memory release upon feature dismissal. For interface construction, Tinder created Layout, a domain-specific Auto Layout wrapper offering declarative syntax for UIKit views to eliminate storyboard merge conflicts while preserving native capabilities. The resulting stack enables isolated testing of business logic, native reactive event binding, and incremental adoption of SwiftUI across the codebase.


### [Image replacement in Canva designs using reverse image search](https://yomu.fyi/post/image-replacement-in-canva-designs-using-reverse-image-search.md)
- Company: [Canva](https://yomu.fyi/company/canva.md)
- Author: Sam Jacobs
- Published: Jan 28, 2025

Canva needed an automated way to replace media in design templates, such as when third-party licensing partnerships expire across more than 150 million images. Existing recommendation engines, perceptual hashing, and text metadata searches failed to capture visual similarity hierarchies or ensure replacement relevance. To build a reverse image search system, engineers evaluated embedding models including CLIP, ViTMAE, DreamSim, CaiT, and DINOv2 alongside an external vector database supporting metadata filtering. Evaluation on sample datasets identified DINOv2 as the best model for preserving subjects, background context, and color tones in photos. Integrated into the Template Assistant as a human-in-the-loop tool, the automated suggestions increased image replacement speeds by 4.5 times during initial pilot testing.


### [The foundations of Canva’s continuous data platform with Snowpipe Streaming](https://yomu.fyi/post/the-foundations-of-canva-s-continuous-data-platform-with-snowpipe-stre.md)
- Company: [Canva](https://yomu.fyi/company/canva.md)
- Author: Jack Caperon
- Published: Jan 6, 2025

As Canva expanded to over 200 million monthly active users, ingestion throughput reached 25 billion records daily, causing AWS Data Firehose costs to consume nearly half of the product analytics platform budget. To reduce these expenses and eliminate intermediate file staging, the engineering team integrated Snowflake's Snowpipe Streaming directly with their Java-based Kinesis Data Streams pipeline. The architecture streams records directly into Snowflake tables using logical channels with offset checkpoints while configuring client buffering up to a five-minute maximum lag. In handling Kinesis Client Library edge cases, record processors coordinate with the LeaseCoordinator to drop leases when channels become unhealthy. Operating in production for over six months, the system ingested more than 20.35 petabytes of data, decreased query latency to roughly ten minutes, and reduced overall cloud spend by 45 percent.


### [Embracing passwordless authentication with Grab’s Passkey](https://yomu.fyi/post/embracing-passwordless-authentication-with-grab-s-passkey.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Ocean Nguyen
- Published: Dec 26, 2024

Grab introduced Passkey to replace vulnerable traditional passwords and cumbersome multi-factor methods with a seamless, phishing-resistant alternative based on the FIDO standard. The architecture relies on an authenticator located on the user's device, a frontend client, and a backend storing only public keys and metadata. During registration and login, the frontend invokes WebAuthn APIs such as navigator.credentials.create and navigator.credentials.get using server-generated challenges to prevent replay attacks. Passkeys synchronize across ecosystems via Google Password Manager and Apple iCloud Keychain, allowing users to authorize logins with their device lock screen. This implementation improves user experience, eliminates the need to store secrets in backend databases, and cuts third-party communication costs associated with OTP delivery.


### [Turbocharging GrabUnlimited with Temporal](https://yomu.fyi/post/turbocharging-grabunlimited-with-temporal.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Michel Parreno
- Published: Dec 12, 2024

GrabUnlimited experienced scaling bottlenecks, corrupted membership states, and elevated production incidents after its subscriber base grew by over 1000%. The original architecture relied on Amazon SQS state machines, 5-minute Redis locks, and daily batch cron jobs that overwhelmed the database and lacked granular idempotency during upstream retries. To eliminate these failure modes, the engineering team migrated the core membership lifecycle to Temporal's workflow orchestration engine. Replacing batch cron jobs with Temporal Timers distributed renewal operations throughout the day, while matching workflow IDs prevented race conditions between renewals and cancellations. This architectural transition resolved database bottlenecks and yielded an 80% reduction in open production incidents.


### [The science of routing print orders](https://yomu.fyi/post/the-science-of-routing-print-orders.md)
- Company: [Canva](https://yomu.fyi/company/canva.md)
- Author: Constantinos Kavadias
- Published: Dec 10, 2024

Canva's global print network requires selecting optimal suppliers to balance delivery times, packaging counts, and environmental emissions. To resolve these challenges before user checkout, the engineering team designed a modular routing architecture that decouples graph construction, decision logic, and path traversal. During graph traversal, the system generates action objects capturing forward paths and decision query results, which are compiled into timestamped routing logs in blob storage for asynchronous auditing. Utilizing preprocessed graph queries alongside ElastiCache, Redis, and database read replicas, the infrastructure sustains high-throughput evaluation without coupling cost logic to traversal code. As a result, print routing completes within an average of 50 milliseconds at the 99th percentile during peak usage while maintaining 99.999% data availability.


### [How we seamlessly migrated high volume real-time streaming traffic from one service to another with zero data loss and duplication](https://yomu.fyi/post/how-we-seamlessly-migrated-high-volume-real-time-streaming-traffic-fro.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Md Riyadh
- Published: Dec 5, 2024

Grab split a backend service's read and write functionalities into separate services to allow independent scaling. Migrating the write path required transferring processing from 16 source Kafka streams—averaging 20,000 reads per second into DynamoDB tables and output streams—with zero data loss or duplication. Standard feature flags were ruled out because rollout propagation delays could introduce minutes of duplicate or missing data during flag toggling. Instead, engineers extracted processing logic into a shared monorepo commons package that used coordinated timestamps to trigger simultaneous cutovers across both services. Temporary validation sinks verified processing accuracy in production prior to the cutover, completing the stream-by-stream migration across three weeks without downtime.


### [How we reduced initialisation time of Product Configuration Management SDK](https://yomu.fyi/post/how-we-reduced-initialisation-time-of-product-configuration-management.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Ram Dilip Pradhan
- Published: Nov 22, 2024

GrabX operates as Grab's central platform for product configuration management, where client services fetch configuration data via an eventually consistent SDK. Services handling around 400 MB of configuration data experienced startup cold starts taking approximately four minutes, creating service stress during traffic spikes. The engineering team resolved this bottleneck through a multi-phase optimization of how the SDK retrieves data from AWS S3. First, sequential downloads of common and service-specific datasets were replaced with concurrent fetching. Next, concurrent downloading and memory loading were applied across large configurations within subscribed services, followed by the complete removal of an outdated disk-caching fallback mechanism. Benchmarks across diverse configuration payloads showed an overall initialisation time reduction of up to 90%.


### [How we reduced peak memory and CPU usage of the product configuration management SDK](https://yomu.fyi/post/how-we-reduced-peak-memory-and-cpu-usage-of-the-product-configuration.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Ram Dilip Pradhan
- Published: Oct 30, 2024

Grab's central product configuration management platform, GrabX, previously aggregated all configurations across every backend service into a single JSON file hosted on AWS S3. Every minute, client SDKs fetched, parsed, and loaded this growing file—which exceeded 100MB—causing CPU throttling spikes, elevated P99 latency, and unnecessary memory consumption. Analysis revealed that 98% of services required less than 1% of the total configuration data. To resolve these bottlenecks, the team partitioned data by service, split configurations into separate S3 files under distinct prefixes, and introduced a per-service changelog for incremental updates. Benchmarks showed the redesign decreased maximum CPU utilisation by over 50% and reduced memory usage by up to 70%.


### [Improving Compute Sustainability: A Case Study](https://yomu.fyi/post/improving-compute-sustainability-a-case-study.md)
- Company: [Two Sigma](https://yomu.fyi/company/two-sigma.md)
- Author: Emily Majewski
- Published: Oct 7, 2024

Two Sigma's large computing footprint drives significant energy consumption and carbon emissions, particularly across live trading applications that require continuous real-time market data caching. To address this overhead without sacrificing performance, engineering teams used a routine hardware refresh to transition from legacy single-process machines to denser multi-core server configurations. By replacing roughly 60 legacy hosts with 28-core processor hardware, the team distributed baseline power draw over more cores and eliminated underutilized compute capacity. This architectural consolidation reduced absolute power consumption across production hosts by 66%, dropping electricity usage from 27 MWh in January 2023 to 10 MWh in January 2024. The initiative subsequently established an annual sustainability rationalization practice for hardware budgeting across the organization.


### [Evolution of Catwalk: Model serving platform at Grab](https://yomu.fyi/post/evolution-of-catwalk-model-serving-platform-at-grab.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Vishal Sharma
- Published: Oct 1, 2024

Grab developed and scaled Catwalk, an internal machine learning model serving platform, to address operational bottlenecks, low resource utilization, and deployment friction between data scientists and backend engineers. The platform transitioned from an admin-managed TensorFlow Serving setup into a low-code self-service system supporting PyTorch and ONNX, before replacing complex Helm charts with Kubernetes Custom Resource Definitions for declarative, blue-green deployment orchestration. To support complex business workflows and multi-model applications, Grab subsequently introduced Catwalk Orchestrator with bundled deployments that allow individual services to scale independently. Across two years, the orchestrator architecture expanded to 200 deployed applications serving approximately 1,400 production machine learning models.


[Newer posts](https://yomu.fyi/topic/architecture/page/10.md) · [Older posts](https://yomu.fyi/topic/architecture/page/12.md)