# Performance > 78 posts about Performance, summarised, each linking to the original. ## Articles ### [Scaling Airbnb’s identity graph with a unified knowledge graph infrastructure](https://yomu.fyi/post/scaling-airbnb-s-identity-graph-with-a-unified-knowledge-graph-infrast.md) - Company: [Airbnb](https://yomu.fyi/company/airbnb.md) - Author: Lucen Zhao - Published: May 19, 2026 Airbnb migrated its massive identity graph from a third-party graph database vendor to a unified, in-house knowledge graph platform to solve long-tail latency and scaling bottlenecks. Built on JanusGraph with AWS DynamoDB for persistence and OpenSearch for indexing, the new architecture decouples storage operations from graph traversal logic. The migration improved P99 query latency, eliminated routine instance reboots, and supported a tenfold increase in write throughput. ### [Scaling developer experience: How we improved Android Studio in a large monorepo](https://yomu.fyi/post/scaling-developer-experience-how-we-improved-android-studio-in-a-large.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Rifqi Fahmi - Published: May 15, 2026 Grab's passenger Android monorepo contains roughly 2,000 modules and 11 million lines of code, leading to IDE sync times exceeding 35 minutes on cold starts. While Bazel handles application builds, Android Studio relies on Gradle sync to construct its project model, requiring Gradle to evaluate every module during configuration. To solve this bottleneck, the team built a custom Focus plugin that restricts IDE syncing to target modules and their direct dependencies. The architecture reuses a static dependency graph generated during existing Gradle-to-Bazel migration tasks to eliminate the Gradle configuration phase during module selection. An IntelliJ extension and a Groovy method-shadowing technique in settings.gradle allow developers to configure focused modules directly through a graphical user interface. ### [From latency to instant: Modernizing GitHub Issues navigation performance](https://yomu.fyi/post/from-latency-to-instant-modernizing-github-issues-navigation-performan.md) - Company: [Github](https://yomu.fyi/company/github.md) - Author: Natalie Guevara - Published: May 14, 2026 GitHub Issues addressed navigation latency by shifting workloads to the client using a local-first, stale-while-revalidate architecture. To evaluate perceived delays, the team tracked user transitions through Highest Priority Content thresholds, aiming for instant rendering in under 200 milliseconds. The core implementation added a persistent client-side cache using IndexedDB, paired with a synchronous in-memory tier to serve hot issue payloads without asynchronous overhead. A selective preheating mechanism resolves missing cache entries for high-intent links without overburdening backend capacity with redundant requests. Following broad rollout, the proportion of instant React soft navigations increased from 4% to approximately 22%, yielding an overall cache-hit ratio of around 33%. ### [Improving storage efficiency in Magic Pocket, our immutable blob store](https://yomu.fyi/post/improving-storage-efficiency-in-magic-pocket-our-immutable-blob-store.md) - Company: [Dropbox](https://yomu.fyi/company/dropbox.md) - Author: Ilya Yakovlev,Andrew Cheung,Binoy Dash,Simran Jumani,Dmitriy Meyerzon,Mark Breitenbach,Ishan Mishra,Kazuaki Okumura,Mike White,Kevin Altschuler,Facundo Agriel - Published: Apr 2, 2026 After introducing a direct erasure-coding service, Dropbox encountered severe storage fragmentation in its immutable blob store, Magic Pocket, due to a large number of under-filled volumes. Because their legacy compaction strategy was only designed to top off near-full volumes, the team created new multi-strategy compaction methods to aggressively consolidate sparse volumes. These approaches leverage dynamic programming and streaming pipelines to rapidly reclaim unused disk capacity at exabyte scale. ### [Reducing our monorepo size to improve developer velocity](https://yomu.fyi/post/reducing-our-monorepo-size-to-improve-developer-velocity.md) - Company: [Dropbox](https://yomu.fyi/company/dropbox.md) - Author: Ilya Yakovlev,Andrew Cheung,Binoy Dash,Simran Jumani,Dmitriy Meyerzon,Mark Breitenbach,Ishan Mishra,Kazuaki Okumura,Mike White,Kevin Altschuler,Facundo Agriel,Ishan Mishra - Published: Mar 25, 2026 Dropbox reduced its core server monorepo size from 87GB to 20GB after approaching GitHub's 100GB limit and suffering from hour-long clone times. The excessive bloat was traced to Git's delta compression heuristics interacting poorly with internationalization directory paths, generating inefficient diffs across different languages. By collaborating with GitHub to run an aggressive server-side repack with tuned window and depth settings, Dropbox cut clone times to under 15 minutes. ### [How we optimized Dash's relevance judge with DSPy](https://yomu.fyi/post/how-we-optimized-dash-s-relevance-judge-with-dspy.md) - Company: [Dropbox](https://yomu.fyi/company/dropbox.md) - Author: Ilya Yakovlev,Andrew Cheung,Binoy Dash,Simran Jumani,Dmitriy Meyerzon,Mark Breitenbach,Ishan Mishra,Kazuaki Okumura,Mike White,Kevin Altschuler,Facundo Agriel,Ishan Mishra,Eric Wang,Dmitriy Meyerzon - Published: Mar 17, 2026 Dropbox Dash optimized its LLM-as-a-judge relevance scoring system using DSPy to migrate from expensive proprietary models to cheaper open-weight alternatives. By establishing automated feedback loops based on human agreement and strict JSON format validation, the team systematically generated robust prompts for new models. This reduced human-score disagreement by 45% and slashed model adaptation time from weeks to days while enabling 10x to 100x more data labeling. ### [Enabling R8 optimization at scale with AI-assisted debugging](https://yomu.fyi/post/enabling-r8-optimization-at-scale-with-ai-assisted-debugging.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Nguyen Van Minh - Published: Mar 12, 2026 Grab experienced widespread Application Not Responding spikes across its Android superapp, driven by memory pressure and complex Jetpack Compose layouts embedded in legacy code. While switching to advanced R8 optimization promised significant performance gains, obfuscated stack traces and two-hour remote compilation cycles stalled investigation across nine million lines of code. To resolve this, engineers built Model Context Protocol tools to automate APK decompilation, deobfuscation, and code context extraction. The team paired these tools with an AI workflow that used the GitLab CLI to generate multiple solution branches and run verification builds in parallel. This strategy replaced hours of manual reverse engineering with minutes of automated analysis, allowing the team to debug and validate aggressive optimizations at scale. ### [Reclaiming Terabytes: Optimizing Android image caching with TLRU](https://yomu.fyi/post/reclaiming-terabytes-optimizing-android-image-caching-with-tlru.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Nguyen Van Minh - Published: Mar 6, 2026 The Grab Android application relies heavily on dynamically downloaded images managed through Glide's disk cache, which originally enforced a 100 MB limit without evaluating content age. While the 90th percentile of users consistently hit this maximum capacity, users below the threshold retained stale promotional and feature assets indefinitely. To reclaim storage while protecting user experience, engineers extended Glide's existing DiskLruCache implementation into a Time-Aware Least Recently Used (TLRU) cache. The TLRU design pairs standard maximum size limits with Time-To-Live eviction based on last-accessed timestamps, alongside a minimum size safety threshold that preserves essential files for returning users. This hybrid approach enables proactive eviction of outdated content without causing empty-cache performance penalties or increasing backend requests. ### [Docker lazy loading at Grab: Accelerating container startup times](https://yomu.fyi/post/docker-lazy-loading-at-grab-accelerating-container-startup-times.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Huong Vuong - Published: Jan 21, 2026 Grab addressed slow cold starts and auto-scaling bottlenecks caused by large container images across data platforms like Airflow and Spark Connect. To eliminate the requirement of downloading complete images before launching, the team evaluated Docker lazy loading via remote snapshotters using eStargz and Seekable OCI (SOCI). Unlike eStargz, which modifies image layers and increased application startup delays during benchmarks, SOCI stores index metadata separately as OCI Artifacts without altering image digests. In production on Amazon EKS, tuning SOCI concurrency and chunk parameters reduced fresh-node image download times by 60%, ultimately delivering a 30% to 40% reduction in P95 startup times. ### [Grab's Mac Cloud Exit supercharges macOS CI/CD](https://yomu.fyi/post/grab-s-mac-cloud-exit-supercharges-macos-ci-cd.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Madushan Gamage - Published: Nov 6, 2025 Scaling iOS CI/CD workloads across multiple large mobile applications led to substantial expenses and queue delays under rented cloud macOS infrastructure. To mitigate network latency to Git servers and cut compute overhead, Grab relocated its build fleet from a US cloud provider to a colocation data center in Malaysia. The infrastructure deployment relies on over 200 bare-metal Mac minis across 42RU racks, purposefully avoiding virtualization solutions that introduced performance and stability penalties. Phased migration and guardrail monitoring verified cluster stability while achieving 20% to 40% reductions in pipeline and app build times. Operating the regional bare-metal cluster over an estimated three-year hardware replacement lifecycle is projected to yield 2.4 million USD in total cost of ownership savings. ### [How we built a custom vision LLM to improve document processing at Grab](https://yomu.fyi/post/how-we-built-a-custom-vision-llm-to-improve-document-processing-at-gra.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Jia Chen - Published: Nov 4, 2025 Document processing for identity verification across Southeast Asia presents challenges due to varied layouts and non-Latin scripts. Traditional OCR and off-the-shelf vision models struggle with accuracy, high latency, or lack of regional language training data. Grab addressed this by creating synthetic regional datasets, using an automated labeling pipeline named Documint, and evaluating open-source multimodal architectures. After initial LoRA fine-tuning failed on complex scripts like Thai and Vietnamese, full-parameter fine-tuning of Qwen2-VL 2B yielded substantial gains. To optimize deployment costs and latency, the team constructed a custom 1B parameter model pairing a Qwen2-VL vision encoder with a Qwen2.5 0.5B language decoder, achieving performance within 3 percentage points of the 2B model at significantly lower latency. ### [Modernising Grab’s model serving platform with NVIDIA Triton Inference Server](https://yomu.fyi/post/modernising-grab-s-model-serving-platform-with-nvidia-triton-inference.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Daniel Tai - Published: Oct 21, 2025 Grab's machine learning serving platform, Catwalk, experienced mounting technical debt, elevated latency, and rising costs from maintaining disparate inference engines for various frameworks. To address these limitations, the team adopted NVIDIA Triton Inference Server to establish a unified engine, starting with an ONNX migration. They built a custom Triton manager component featuring a proxy layer to translate legacy API requests alongside a server manager that handles model downloads, verification, configuration, and health checks. Within ten days, over half of online deployments migrated seamlessly without requiring client code modifications. The transition reduced p90 latency from 120ms to 20ms on large transformer models and cut average infrastructure spend by approximately 20 percent across evaluated services. ### [Highly concurrent in-memory counter in GoLang](https://yomu.fyi/post/highly-concurrent-in-memory-counter-in-golang.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Naveen Kumar Jakuva Premkumar - Published: Oct 6, 2025 High database CPU utilization caused by relentless write traffic for marketing campaign counts prompted Grab to build an in-memory aggregation buffer rather than migrate from SQL to NoSQL. The team implemented an in-memory counter in Go that batches usage updates and periodically flushes them to persistent storage. To eliminate the serialization bottlenecks of mutex-locked maps under high concurrency, the design adopted Go's sync.Map using atomic CompareAndSwap retries for value updates alongside LoadAndDelete for periodic background flushes. Because a finite set of campaign keys is repeatedly accessed, operations hit the internal read map fast path nearly 99% of the time, achieving a threefold throughput improvement over standard mutex locks in benchmarks. In production, this architecture decreased database update queries by 68% and reduced master database CPU utilization from 35% to 18%. ### [Powering Partner Gateway metrics with Apache Pinot](https://yomu.fyi/post/powering-partner-gateway-metrics-with-apache-pinot.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Alvis Chew - Published: Sep 23, 2025 Grab needed to power real-time analytics dashboards for its Partner Gateway, tracking API status codes and latency across datasets reaching 6.8 billion rows over 30-day windows. Initial aggregation queries on datasets exceeding 150GB frequently timed out past 10 seconds, failing to meet the platform's 300-millisecond service level agreement. To support low-latency Online Analytical Processing queries, Grab routed metric streams through Apache Kafka and Apache Flink into Apache Pinot. Query execution was then accelerated by partitioning Kafka topics by metric name, adding rounded time interval columns, and implementing Star-tree indexes for multidimensional pre-aggregation. ### [Taming the monorepo beast: Our journey to a leaner, faster GitLab repo](https://yomu.fyi/post/taming-the-monorepo-beast-our-journey-to-a-leaner-faster-gitlab-repo.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Nagendra Gangwar - Published: Sep 16, 2025 Grab's decade-old Go monorepo grew to 12.7 million commits and 250GB of Git data, causing Gitaly replication delays of up to four minutes that routed all read traffic exclusively to the primary node and slowed developer operations. After staging tests proved that shallow history reduced replication lag from hundreds of seconds to under three seconds, standard rewriting tools like git filter-repo and git rebase failed due to complex merge histories and repository scale. To overcome runner memory limits and lengthy git garbage collection cycles, the engineering team implemented a custom two-phase migration script. The script selectively migrated 2,000+ critical dependency tags and one month of recent history, flattening merge commits, embedding legacy hashes for traceability, and reducing total commit volume by 99.9%. ### [Counter Service: How we rewrote it in Rust](https://yomu.fyi/post/counter-service-how-we-rewrote-it-in-rust.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Jia Long Loh - Published: Jun 20, 2025 The Integrity Data Platform team rewrote Counter Service, a high-throughput Golang microservice serving event counts for fraud rules and machine learning models, to evaluate the operational return on investment of Rust. Rather than performing a line-by-line translation, engineers approached the service as a black box, reimplementing core application logic from scratch to satisfy established gRPC contracts across Scylla and Redis. The team resolved internal Go tooling dependencies by building custom configuration template parsers using the nom parser combinator and selected targeted open-source crates such as fred.rs and Cadence. Adapting to Rust required navigating cooperative, stackless async execution compared to Go's preemptive concurrency model, alongside managing borrow checker constraints. Ultimately, the rewrite achieved a 70% reduction in infrastructure costs while maintaining comparable service performance. ### [Evaluating performance impact of removing Redis-cache from a Scylla-backed service](https://yomu.fyi/post/evaluating-performance-impact-of-removing-redis-cache-from-a-scylla-ba.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Md Riyadh - Published: Apr 11, 2025 Grab operates a high-throughput Rust read service that aggregates counter metrics from Scylla tables across minutely, hourly, and daily granularities. The service initially cached aggregated responses in Redis using keys rounded to 15-minute intervals alongside a five-minute TTL. Because incoming queries predominantly requested recent data, transitions between 15-minute windows caused simultaneous cache misses across active configurations, resulting in severe Scylla traffic spikes, latency surges, and timeouts. To resolve the load imbalance, engineers proposed removing the Redis cache entirely and relying directly on Scylla's native internal caching. The rollout was staged in production by deterministically disabling Redis caching for specific counter configurations using mathematical operations on configuration IDs. ### [How we reduced initialisation time of Product Configuration Management SDK](https://yomu.fyi/post/how-we-reduced-initialisation-time-of-product-configuration-management.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Ram Dilip Pradhan - Published: Nov 22, 2024 GrabX operates as Grab's central platform for product configuration management, where client services fetch configuration data via an eventually consistent SDK. Services handling around 400 MB of configuration data experienced startup cold starts taking approximately four minutes, creating service stress during traffic spikes. The engineering team resolved this bottleneck through a multi-phase optimization of how the SDK retrieves data from AWS S3. First, sequential downloads of common and service-specific datasets were replaced with concurrent fetching. Next, concurrent downloading and memory loading were applied across large configurations within subscribed services, followed by the complete removal of an outdated disk-caching fallback mechanism. Benchmarks across diverse configuration payloads showed an overall initialisation time reduction of up to 90%. ### [How we reduced peak memory and CPU usage of the product configuration management SDK](https://yomu.fyi/post/how-we-reduced-peak-memory-and-cpu-usage-of-the-product-configuration.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Ram Dilip Pradhan - Published: Oct 30, 2024 Grab's central product configuration management platform, GrabX, previously aggregated all configurations across every backend service into a single JSON file hosted on AWS S3. Every minute, client SDKs fetched, parsed, and loaded this growing file—which exceeded 100MB—causing CPU throttling spikes, elevated P99 latency, and unnecessary memory consumption. Analysis revealed that 98% of services required less than 1% of the total configuration data. To resolve these bottlenecks, the team partitioned data by service, split configurations into separate S3 files under distinct prefixes, and introduced a per-service changelog for incremental updates. Benchmarks showed the redesign decreased maximum CPU utilisation by over 50% and reduced memory usage by up to 70%. ### [Profile-guided optimisation (PGO) on Grab services](https://yomu.fyi/post/profile-guided-optimisation-pgo-on-grab-services.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Yonghao Hu - Published: Jun 5, 2024 Profile-guided optimisation (PGO) improves Go application performance by feeding production CPU profile data into subsequent compiler builds. Teams evaluated PGO across multiple Grab services, including TalariaDB and an orchestrator within the Catwalk service, using Go 1.20 and Go 1.21 builds. TalariaDB saw substantial resource savings after compiling Docker images with a 360-second pprof profile, resulting in at least a 10% drop in CPU usage, a 30% drop in memory consumption, and a 38% reduction in persistent volume usage for event ingestion queues. Conversely, testing on Catwalk showed only a 5% performance gain after adjusting the profiling duration from 59 seconds to 6 minutes, an outcome deemed insufficient to justify pipeline modifications. Broader adoption across monorepo services remains constrained until the central build process supports PGO arguments and automated profile retrieval. [Newer posts](https://yomu.fyi/topic/performance.md) · [Older posts](https://yomu.fyi/topic/performance/page/3.md)