# Scalability > 84 posts about Scalability, summarised, each linking to the original. ## Articles ### [Sitar-agent: Building a reliable dynamic configuration sidecar at scale](https://yomu.fyi/post/sitar-agent-building-a-reliable-dynamic-configuration-sidecar-at-scale.md) - Company: [Airbnb](https://yomu.fyi/company/airbnb.md) - Author: Bo Teng - Published: Jun 4, 2026 Airbnb modernized sitar-agent, a Kubernetes sidecar that delivers dynamic configuration updates to thousands of polyglot service instances without requiring redeployments. The architecture uses periodic AWS S3 snapshots to enable fast, decoupled pod startups and maintains local file-based storage for the main container to read configs with in-memory caching. Key design decisions include retaining an isolated sidecar over an in-process library and optimizing pull-based polling with server-side caching. ### [Slack AI: The Path to Multi-Cloud](https://yomu.fyi/post/slack-ai-the-path-to-multi-cloud.md) - Company: [Slack](https://yomu.fyi/company/slack.md) - Author: Shaurya Kethireddy - Published: May 28, 2026 Slack evolved its Slack AI serving infrastructure across multiple phases to handle enterprise LLM workloads reliably and securely. The initial deployment on AWS SageMaker provided zero-knowledge escrow VPC isolation and FedRAMP compliance, but engineers faced scaling latency, GPU scarcity, and significant operational overhead. Slack then migrated live traffic to Amazon Bedrock to leverage managed Model Units and eliminate model release lag without customer-facing incidents. However, fixed Provisioned Throughput commitments and regional peak traffic variations created persistent underutilization challenges. Consequently, Slack expanded into a multi-cloud orchestration architecture that normalizes disparate provider APIs, integrates unified cross-cloud telemetry, and routes traffic dynamically around latency spikes and outages. ### [Scaling Airbnb’s identity graph with a unified knowledge graph infrastructure](https://yomu.fyi/post/scaling-airbnb-s-identity-graph-with-a-unified-knowledge-graph-infrast.md) - Company: [Airbnb](https://yomu.fyi/company/airbnb.md) - Author: Lucen Zhao - Published: May 19, 2026 Airbnb migrated its massive identity graph from a third-party graph database vendor to a unified, in-house knowledge graph platform to solve long-tail latency and scaling bottlenecks. Built on JanusGraph with AWS DynamoDB for persistence and OpenSearch for indexing, the new architecture decouples storage operations from graph traversal logic. The migration improved P99 query latency, eliminated routine instance reboots, and supported a tenfold increase in write throughput. ### [Scaling developer experience: How we improved Android Studio in a large monorepo](https://yomu.fyi/post/scaling-developer-experience-how-we-improved-android-studio-in-a-large.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Rifqi Fahmi - Published: May 15, 2026 Grab's passenger Android monorepo contains roughly 2,000 modules and 11 million lines of code, leading to IDE sync times exceeding 35 minutes on cold starts. While Bazel handles application builds, Android Studio relies on Gradle sync to construct its project model, requiring Gradle to evaluate every module during configuration. To solve this bottleneck, the team built a custom Focus plugin that restricts IDE syncing to target modules and their direct dependencies. The architecture reuses a static dependency graph generated during existing Gradle-to-Bazel migration tasks to eliminate the Gradle configuration phase during module selection. An IntelliJ extension and a Groovy method-shadowing technique in settings.gradle allow developers to configure focused modules directly through a graphical user interface. ### [Improving storage efficiency in Magic Pocket, our immutable blob store](https://yomu.fyi/post/improving-storage-efficiency-in-magic-pocket-our-immutable-blob-store.md) - Company: [Dropbox](https://yomu.fyi/company/dropbox.md) - Author: Ilya Yakovlev,Andrew Cheung,Binoy Dash,Simran Jumani,Dmitriy Meyerzon,Mark Breitenbach,Ishan Mishra,Kazuaki Okumura,Mike White,Kevin Altschuler,Facundo Agriel - Published: Apr 2, 2026 After introducing a direct erasure-coding service, Dropbox encountered severe storage fragmentation in its immutable blob store, Magic Pocket, due to a large number of under-filled volumes. Because their legacy compaction strategy was only designed to top off near-full volumes, the team created new multi-strategy compaction methods to aggressively consolidate sparse volumes. These approaches leverage dynamic programming and streaming pipelines to rapidly reclaim unused disk capacity at exabyte scale. ### [Grab's Mac Cloud Exit supercharges macOS CI/CD](https://yomu.fyi/post/grab-s-mac-cloud-exit-supercharges-macos-ci-cd.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Madushan Gamage - Published: Nov 6, 2025 Scaling iOS CI/CD workloads across multiple large mobile applications led to substantial expenses and queue delays under rented cloud macOS infrastructure. To mitigate network latency to Git servers and cut compute overhead, Grab relocated its build fleet from a US cloud provider to a colocation data center in Malaysia. The infrastructure deployment relies on over 200 bare-metal Mac minis across 42RU racks, purposefully avoiding virtualization solutions that introduced performance and stability penalties. Phased migration and guardrail monitoring verified cluster stability while achieving 20% to 40% reductions in pipeline and app build times. Operating the regional bare-metal cluster over an estimated three-year hardware replacement lifecycle is projected to yield 2.4 million USD in total cost of ownership savings. ### [Machine-learning predictive autoscaling for Flink](https://yomu.fyi/post/machine-learning-predictive-autoscaling-for-flink.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Minh Nhat Nguyen - Published: Oct 30, 2025 Managing resource allocations for expanding Apache Flink stream-processing workloads often leads to over-provisioning or unstable reactive scaling. Reactive autoscaling with Kubernetes Horizontal Pod Autoscaling triggers severe restart spikes because pipelines must reprocess backlogged records from checkpoints, inducing feedback loops and scaling fluctuations. To prevent these spirals, a predictive vertical autoscaling system uses upstream Kafka message throughput as an independent workload metric. The architecture forecasts future workload using time-series models and maps the projected throughput to required TaskManager CPU allocations through a trained regression model. A custom controller vertically scales TaskManager CPU resources before traffic changes arrive, avoiding restart-induced latency spikes. ### [Highly concurrent in-memory counter in GoLang](https://yomu.fyi/post/highly-concurrent-in-memory-counter-in-golang.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Naveen Kumar Jakuva Premkumar - Published: Oct 6, 2025 High database CPU utilization caused by relentless write traffic for marketing campaign counts prompted Grab to build an in-memory aggregation buffer rather than migrate from SQL to NoSQL. The team implemented an in-memory counter in Go that batches usage updates and periodically flushes them to persistent storage. To eliminate the serialization bottlenecks of mutex-locked maps under high concurrency, the design adopted Go's sync.Map using atomic CompareAndSwap retries for value updates alongside LoadAndDelete for periodic background flushes. Because a finite set of campaign keys is repeatedly accessed, operations hit the internal read map fast path nearly 99% of the time, achieving a threefold throughput improvement over standard mutex locks in benchmarks. In production, this architecture decreased database update queries by 68% and reduced master database CPU utilization from 35% to 18%. ### [Powering Partner Gateway metrics with Apache Pinot](https://yomu.fyi/post/powering-partner-gateway-metrics-with-apache-pinot.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Alvis Chew - Published: Sep 23, 2025 Grab needed to power real-time analytics dashboards for its Partner Gateway, tracking API status codes and latency across datasets reaching 6.8 billion rows over 30-day windows. Initial aggregation queries on datasets exceeding 150GB frequently timed out past 10 seconds, failing to meet the platform's 300-millisecond service level agreement. To support low-latency Online Analytical Processing queries, Grab routed metric streams through Apache Kafka and Apache Flink into Apache Pinot. Query execution was then accelerated by partitioning Kafka topics by metric name, adding rounded time interval columns, and implementing Star-tree indexes for multidimensional pre-aggregation. ### [Taming the monorepo beast: Our journey to a leaner, faster GitLab repo](https://yomu.fyi/post/taming-the-monorepo-beast-our-journey-to-a-leaner-faster-gitlab-repo.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Nagendra Gangwar - Published: Sep 16, 2025 Grab's decade-old Go monorepo grew to 12.7 million commits and 250GB of Git data, causing Gitaly replication delays of up to four minutes that routed all read traffic exclusively to the primary node and slowed developer operations. After staging tests proved that shallow history reduced replication lag from hundreds of seconds to under three seconds, standard rewriting tools like git filter-repo and git rebase failed due to complex merge histories and repository scale. To overcome runner memory limits and lengthy git garbage collection cycles, the engineering team implemented a custom two-phase migration script. The script selectively migrated 2,000+ critical dependency tags and one month of recent history, flattening merge commits, embedding legacy hashes for traceability, and reducing total commit volume by 99.9%. ### [Data mesh at Grab part I: Building trust through certification](https://yomu.fyi/post/data-mesh-at-grab-part-i-building-trust-through-certification.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Chun Rong Phang - Published: Aug 19, 2025 Rapid business growth across multiple verticals led Grab's centralized data engineering model to become an unscalable bottleneck, resulting in duplicate pipelines, ambiguous ownership, and broken downstream dependencies. To resolve these issues, the organization initiated a data mesh journey called Signals Marketplace that decentralizes data management and treats data as a product. A central data certification system establishes formal data contracts covering schemas, SLAs, freshness, and retention, while assigning clear Business Data Owners and Technical Data Owners. Breaches in contract guarantees automatically generate Data Production Incident tickets to enforce accountability and root-cause fixes. Consequently, 75% of internal queries now target certified assets, redundant tables saw a 400% year-over-year deprecation increase, and the total number of top-used datasets dropped by over 58%. ### [Turbocharging GrabUnlimited with Temporal](https://yomu.fyi/post/turbocharging-grabunlimited-with-temporal.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Michel Parreno - Published: Dec 12, 2024 GrabUnlimited experienced scaling bottlenecks, corrupted membership states, and elevated production incidents after its subscriber base grew by over 1000%. The original architecture relied on Amazon SQS state machines, 5-minute Redis locks, and daily batch cron jobs that overwhelmed the database and lacked granular idempotency during upstream retries. To eliminate these failure modes, the engineering team migrated the core membership lifecycle to Temporal's workflow orchestration engine. Replacing batch cron jobs with Temporal Timers distributed renewal operations throughout the day, while matching workflow IDs prevented race conditions between renewals and cancellations. This architectural transition resolved database bottlenecks and yielded an 80% reduction in open production incidents. ### [How we reduced peak memory and CPU usage of the product configuration management SDK](https://yomu.fyi/post/how-we-reduced-peak-memory-and-cpu-usage-of-the-product-configuration.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Ram Dilip Pradhan - Published: Oct 30, 2024 Grab's central product configuration management platform, GrabX, previously aggregated all configurations across every backend service into a single JSON file hosted on AWS S3. Every minute, client SDKs fetched, parsed, and loaded this growing file—which exceeded 100MB—causing CPU throttling spikes, elevated P99 latency, and unnecessary memory consumption. Analysis revealed that 98% of services required less than 1% of the total configuration data. To resolve these bottlenecks, the team partitioned data by service, split configurations into separate S3 files under distinct prefixes, and introduced a per-service changelog for incremental updates. Benchmarks showed the redesign decreased maximum CPU utilisation by over 50% and reduced memory usage by up to 70%. ### [Evolution of Catwalk: Model serving platform at Grab](https://yomu.fyi/post/evolution-of-catwalk-model-serving-platform-at-grab.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Vishal Sharma - Published: Oct 1, 2024 Grab developed and scaled Catwalk, an internal machine learning model serving platform, to address operational bottlenecks, low resource utilization, and deployment friction between data scientists and backend engineers. The platform transitioned from an admin-managed TensorFlow Serving setup into a low-code self-service system supporting PyTorch and ONNX, before replacing complex Helm charts with Kubernetes Custom Resource Definitions for declarative, blue-green deployment orchestration. To support complex business workflows and multi-model applications, Grab subsequently introduced Catwalk Orchestrator with bundled deployments that allow individual services to scale independently. Across two years, the orchestrator architecture expanded to 200 deployed applications serving approximately 1,400 production machine learning models. ### [Chimera Sandbox: A scalable experimentation and development platform for Notebook services](https://yomu.fyi/post/chimera-sandbox-a-scalable-experimentation-and-development-platform-fo.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Arkar Min Aung - Published: Aug 27, 2024 Grab's Chimera team built Chimera Sandbox to address resource accessibility challenges and enable rapid machine learning experimentation across technical and non-technical teams. The platform runs on multi-cluster Kubernetes infrastructure capable of scaling to thousands of nodes per cluster during peak loads. Users can provision customizable Docker environments with varying CPU and GPU configurations, collaborate in real time, and query data using Spark-on-Kubernetes integrations and helper magic cells. The workspace integrates with Grab's AI Gateway for LLM access alongside tools like MLflow, Optuna, RStudio, and LabelStudio on shared storage, backed by user-level cost attribution. ### [LLM-powered data classification for data entities at scale](https://yomu.fyi/post/llm-powered-data-classification-for-data-entities-at-scale.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Hualin Liu - Published: Jul 15, 2024 Grab needed to classify sensitive data at the table and column level across petabytes of database tables and streaming schemas. Manual schema-tiering campaigns had resulted in half of all schemas receiving overly strict Tier 1 access controls, while an initial automated service using regex patterns and third-party machine learning produced high false-positive rates and lacked customizability. To address this, the Caspian data engineering and governance teams enhanced their internal orchestration service, Gemini, by integrating GPT-3.5 via Azure OpenAI. Gemini aggregates classification requests into mini-batches, handles API rate limits, and uses prompt engineering—including few-shot examples, curated tag libraries, and explicit JSON DTO schemas—to reliably tag columns for data owner verification. ### [How we evaluated the business impact of marketing campaigns](https://yomu.fyi/post/how-we-evaluated-the-business-impact-of-marketing-campaigns.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Jie Zhang - Published: May 23, 2024 Grab needed a scalable, centralised way to evaluate the business impact of millions of marketing messages sent across expanding campaigns. Manual analysis by marketing analysts produced inconsistent results and failed to scale, leading to the creation of multi-touch attribution and controlled experiment methodologies. The team designed a Marketing attribution model (MAM) that credits touchpoints based on relevance and recency, while using stratified hold-out groups to assess single campaign lift against baseline metrics. Statistical significance calculations, implemented using the jStat library based on the central limit theorem, ensure observed business metric differences like gross merchandise value are not random. The system provides campaign managers with direct access to impact analyses while the team works on challenges surrounding smaller sample sizes and hold-out group constraints. ### [Managing dynamic marketplace content at scale: Grab's approach to content moderation](https://yomu.fyi/post/managing-dynamic-marketplace-content-at-scale-grab-s-approach-to-conte.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Poonam Gambhire - Published: Feb 1, 2024 Grab manages content moderation across GrabFood and GrabMart, where merchants update over 100,000 item listings daily across eight countries. The platform must adhere to internal rules, local government regulations, and external platform guidelines across these markets. To handle scale and regional nuance, Grab uses an in-house automated system powered by algorithms and machine learning to scan daily listings for violations. Items flagged as requiring subjective judgment or cultural awareness are routed to human moderators for manual review. Additionally, Grab collaborates with Google to maintain compliance with Play Store policies. ### [Sliding window rate limits in distributed systems](https://yomu.fyi/post/sliding-window-rate-limits-in-distributed-systems.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Naveen Kumar Jakuva Premkumar - Published: Dec 14, 2023 Marketing communications across Grab's user base risked causing notification overload and consent revocations. To enforce personalized daily and weekly frequency caps across more than 270 million users, the team addressed segment membership storage and communication rate limiting. They adopted roaring bitmaps instead of Bloom filters to compactly store user segment data without hash collisions or costly rebuilds upon deletion. For frequency capping, they chose Amazon ElastiCache for Redis over DynamoDB, executing a sliding log rate limiting algorithm directly on the cluster using Lua scripts and sorted sets. Timestamps are stored as sorted set scores, and historical data is cleaned up via eviction ranges to prevent unbounded memory growth. ### [Road localisation in GrabMaps](https://yomu.fyi/post/road-localisation-in-grabmaps.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Roxana Crisan - Published: Nov 17, 2023 Grab needed to localise nearly 30 million road segments across more than 10,000 area boundaries in Southeast Asia to attach hyperlocal attributes like language, driving side, and vehicle access rules. Testing direct geometric inclusion between complex road polylines and boundary polygons proved computationally prohibitive for daily map generation pipelines. To optimize this process, Grab introduced geohashes as rectangular proxies, precomputing geohash coverage across borders and road segments before joining the datasets in parallel. To resolve misclassification errors near borders without sacrificing performance, the team evaluated geohash coverage percentages and refined boundary-touching geohashes with polygon clipping. A laptop benchmark on a subset of data reduced runtime from 38 minutes with geometric intersection to 78 seconds using the geohash approximation. [Newer posts](https://yomu.fyi/topic/scalability.md) · [Older posts](https://yomu.fyi/topic/scalability/page/3.md)