# Performance > 85 posts about Performance, summarised, each linking to the original. ## Articles ### [Powering Partner Gateway metrics with Apache Pinot](https://yomu.fyi/post/powering-partner-gateway-metrics-with-apache-pinot.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Alvis Chew - Published: Sep 23, 2025 Grab needed to power real-time analytics dashboards for its Partner Gateway, tracking API status codes and latency across datasets reaching 6.8 billion rows over 30-day windows. Initial aggregation queries on datasets exceeding 150GB frequently timed out past 10 seconds, failing to meet the platform's 300-millisecond service level agreement. To support low-latency Online Analytical Processing queries, Grab routed metric streams through Apache Kafka and Apache Flink into Apache Pinot. Query execution was then accelerated by partitioning Kafka topics by metric name, adding rounded time interval columns, and implementing Star-tree indexes for multidimensional pre-aggregation. ### [Taming the monorepo beast: Our journey to a leaner, faster GitLab repo](https://yomu.fyi/post/taming-the-monorepo-beast-our-journey-to-a-leaner-faster-gitlab-repo.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Nagendra Gangwar - Published: Sep 16, 2025 Grab's decade-old Go monorepo grew to 12.7 million commits and 250GB of Git data, causing Gitaly replication delays of up to four minutes that routed all read traffic exclusively to the primary node and slowed developer operations. After staging tests proved that shallow history reduced replication lag from hundreds of seconds to under three seconds, standard rewriting tools like git filter-repo and git rebase failed due to complex merge histories and repository scale. To overcome runner memory limits and lengthy git garbage collection cycles, the engineering team implemented a custom two-phase migration script. The script selectively migrated 2,000+ critical dependency tags and one month of recent history, flattening merge commits, embedding legacy hashes for traceability, and reducing total commit volume by 99.9%. ### [Counter Service: How we rewrote it in Rust](https://yomu.fyi/post/counter-service-how-we-rewrote-it-in-rust.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Jia Long Loh - Published: Jun 20, 2025 The Integrity Data Platform team rewrote Counter Service, a high-throughput Golang microservice serving event counts for fraud rules and machine learning models, to evaluate the operational return on investment of Rust. Rather than performing a line-by-line translation, engineers approached the service as a black box, reimplementing core application logic from scratch to satisfy established gRPC contracts across Scylla and Redis. The team resolved internal Go tooling dependencies by building custom configuration template parsers using the nom parser combinator and selected targeted open-source crates such as fred.rs and Cadence. Adapting to Rust required navigating cooperative, stackless async execution compared to Go's preemptive concurrency model, alongside managing borrow checker constraints. Ultimately, the rewrite achieved a 70% reduction in infrastructure costs while maintaining comparable service performance. ### [Evaluating performance impact of removing Redis-cache from a Scylla-backed service](https://yomu.fyi/post/evaluating-performance-impact-of-removing-redis-cache-from-a-scylla-ba.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Md Riyadh - Published: Apr 11, 2025 Grab operates a high-throughput Rust read service that aggregates counter metrics from Scylla tables across minutely, hourly, and daily granularities. The service initially cached aggregated responses in Redis using keys rounded to 15-minute intervals alongside a five-minute TTL. Because incoming queries predominantly requested recent data, transitions between 15-minute windows caused simultaneous cache misses across active configurations, resulting in severe Scylla traffic spikes, latency surges, and timeouts. To resolve the load imbalance, engineers proposed removing the Redis cache entirely and relying directly on Scylla's native internal caching. The rollout was staged in production by deterministically disabling Redis caching for specific counter configurations using mathematical operations on configuration IDs. ### [How we reduced initialisation time of Product Configuration Management SDK](https://yomu.fyi/post/how-we-reduced-initialisation-time-of-product-configuration-management.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Ram Dilip Pradhan - Published: Nov 22, 2024 GrabX operates as Grab's central platform for product configuration management, where client services fetch configuration data via an eventually consistent SDK. Services handling around 400 MB of configuration data experienced startup cold starts taking approximately four minutes, creating service stress during traffic spikes. The engineering team resolved this bottleneck through a multi-phase optimization of how the SDK retrieves data from AWS S3. First, sequential downloads of common and service-specific datasets were replaced with concurrent fetching. Next, concurrent downloading and memory loading were applied across large configurations within subscribed services, followed by the complete removal of an outdated disk-caching fallback mechanism. Benchmarks across diverse configuration payloads showed an overall initialisation time reduction of up to 90%. ### [How we reduced peak memory and CPU usage of the product configuration management SDK](https://yomu.fyi/post/how-we-reduced-peak-memory-and-cpu-usage-of-the-product-configuration.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Ram Dilip Pradhan - Published: Oct 30, 2024 Grab's central product configuration management platform, GrabX, previously aggregated all configurations across every backend service into a single JSON file hosted on AWS S3. Every minute, client SDKs fetched, parsed, and loaded this growing file—which exceeded 100MB—causing CPU throttling spikes, elevated P99 latency, and unnecessary memory consumption. Analysis revealed that 98% of services required less than 1% of the total configuration data. To resolve these bottlenecks, the team partitioned data by service, split configurations into separate S3 files under distinct prefixes, and introduced a per-service changelog for incremental updates. Benchmarks showed the redesign decreased maximum CPU utilisation by over 50% and reduced memory usage by up to 70%. ### [Profile-guided optimisation (PGO) on Grab services](https://yomu.fyi/post/profile-guided-optimisation-pgo-on-grab-services.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Yonghao Hu - Published: Jun 5, 2024 Profile-guided optimisation (PGO) improves Go application performance by feeding production CPU profile data into subsequent compiler builds. Teams evaluated PGO across multiple Grab services, including TalariaDB and an orchestrator within the Catwalk service, using Go 1.20 and Go 1.21 builds. TalariaDB saw substantial resource savings after compiling Docker images with a 360-second pprof profile, resulting in at least a 10% drop in CPU usage, a 30% drop in memory consumption, and a 38% reduction in persistent volume usage for event ingestion queues. Conversely, testing on Catwalk showed only a 5% performance gain after adjusting the profiling duration from 59 seconds to 6 minutes, an outcome deemed insufficient to justify pipeline modifications. Broader adoption across monorepo services remains constrained until the central build process supports PGO arguments and automated profile retrieval. ### [Iris - Turning observations into actionable insights for enhanced decision making](https://yomu.fyi/post/iris-turning-observations-into-actionable-insights-for-enhanced-decisi.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Huong Vuong - Published: Apr 3, 2024 Standard infrastructure monitoring tools track CPU and memory usage at the host or container level, obscuring the precise resource consumption of individual Spark jobs on shared instances. To achieve granular observability, the Iris platform gathers job metrics directly at the Java Virtual Machine level without requiring changes to user application code. Data collection relies on Uber's JVM Profiler for deep JVM, thread, and memory pool diagnostics alongside sparkMeasure for stage and task execution statistics. Collected metrics route asynchronously through an Apache Kafka queue to avoid execution bottlenecks, feeding a Telegraf, InfluxDB, and Grafana stack for real-time monitoring while archiving to a data lake for offline analytics. This multi-tiered telemetry enables engineering teams to identify over- or under-provisioned jobs and optimize ETL resource allocations. ### [Android App Size at Scale with Project Bonsai](https://yomu.fyi/post/android-app-size-at-scale-with-project-bonsai.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Nguyen Van Minh - Published: Mar 1, 2024 Grab launched Project Bonsai to optimize the size of its Android superapp, which encompassed over four million lines of code across 1,500 modules. Because large app sizes negatively impact user acquisition on low-end devices with constrained bandwidth, the engineering team structured its optimization strategy around measurement, reduction, and containment. The team developed App Sizer, a custom CI-integrated tool that tracks binary compositions and module size contributions in Grafana. Initial reductions targeted Java and Kotlin dex files, resolving bloated transitive R classes by upgrading the Android Gradle Plugin to eliminate overly broad R8 retention rules. These initiatives achieved a 26% reduction in app download size while simultaneously decreasing overall disk footprint. ### [Enabling near real-time data analytics on the data lake](https://yomu.fyi/post/enabling-near-real-time-data-analytics-on-the-data-lake.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Shi Kai Ng - Published: Feb 23, 2024 Traditional data lake setups using Parquet on Hive metastores struggle with frequent updates and long pipeline intervals, introducing significant latency for ad hoc queries. Grab solved this by implementing Apache Hudi to support near real-time analytics across bounded relational databases and unbounded Kafka streams. For high-throughput sources, Flink streams Avro log files to Merge On Read tables and generates compaction plans for asynchronous Spark writers. Low-throughput workloads leverage Copy On Write tables, while relational database sources ingest binlogs via Flink Change Data Capture connectors. This architecture reduced analytics data latency to the minute level without overloading production databases and Kafka clusters. ### [The journey of building a comprehensive attribution platform](https://yomu.fyi/post/the-journey-of-building-a-comprehensive-attribution-platform.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Kang Huang - Published: Feb 20, 2024 Grab needed to modernize its marketing analytics from manual ad hoc queries and high data latency to a platform supporting real-time attribution for pricing models like cost per order. The engineering team initially deployed a pure stream-processing engine using Kappa architecture, Kafka, ScyllaDB, and Redis, which reduced latency from days to minutes and merged ads and promo touchpoints. However, stream-only processing faced high costs, out-of-order event issues, and difficulties running multi-touch models across longer historical windows. Grab transitioned to a Lambda architecture pairing Coban stream processing with Spark-based batch ETL and Amazon S3. This hybrid design separated real-time operational metrics from historical batch reporting, cutting real-time processing costs by approximately 25% while maintaining under 1% data discrepancy. ### [Sliding window rate limits in distributed systems](https://yomu.fyi/post/sliding-window-rate-limits-in-distributed-systems.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Naveen Kumar Jakuva Premkumar - Published: Dec 14, 2023 Marketing communications across Grab's user base risked causing notification overload and consent revocations. To enforce personalized daily and weekly frequency caps across more than 270 million users, the team addressed segment membership storage and communication rate limiting. They adopted roaring bitmaps instead of Bloom filters to compactly store user segment data without hash collisions or costly rebuilds upon deletion. For frequency capping, they chose Amazon ElastiCache for Redis over DynamoDB, executing a sliding log rate limiting algorithm directly on the cluster using Lua scripts and sorted sets. Timestamps are stored as sorted set scores, and historical data is cleaned up via eviction ranges to prevent unbounded memory growth. ### [Road localisation in GrabMaps](https://yomu.fyi/post/road-localisation-in-grabmaps.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Roxana Crisan - Published: Nov 17, 2023 Grab needed to localise nearly 30 million road segments across more than 10,000 area boundaries in Southeast Asia to attach hyperlocal attributes like language, driving side, and vehicle access rules. Testing direct geometric inclusion between complex road polylines and boundary polygons proved computationally prohibitive for daily map generation pipelines. To optimize this process, Grab introduced geohashes as rectangular proxies, precomputing geohash coverage across borders and road segments before joining the datasets in parallel. To resolve misclassification errors near borders without sacrificing performance, the team evaluated geohash coverage percentages and refined boundary-touching geohashes with polygon clipping. A laptop benchmark on a subset of data reduced runtime from 38 minutes with geometric intersection to 78 seconds using the geohash approximation. ### [Stepping up marketing for advertisers: Scalable lookalike audience](https://yomu.fyi/post/stepping-up-marketing-for-advertisers-scalable-lookalike-audience.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: William Wu - Published: Sep 22, 2023 Grab's legacy lookalike audience generation platform suffered from long creation SLAs of two working days, high costs, and low weekly update frequencies. To resolve these bottlenecks, the engineering team designed an embedding-based platform powered by an in-memory retrieval service and automated update pipelines. The system creates audience representations by averaging constituent passenger embeddings, determining user membership through real-time cosine score thresholds. To eliminate feature store latency while fitting all embeddings into memory, a hash-based compression method cuts passenger embedding storage needs by roughly 90%. Consequently, audience availability dropped to within 15 minutes of campaign creation, audience generation costs fell by 98%, and ad impressions and clicks doubled. ### [Streamlining Grab's Segmentation Platform with faster creation and lower latency](https://yomu.fyi/post/streamlining-grab-s-segmentation-platform-with-faster-creation-and-low.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Jake Ng - Published: Aug 15, 2023 Grab's Segmentation Platform previously stored user-to-segment mappings across individual rows in ScyllaDB, causing write bottlenecks during segment creation and read latencies too high for downstream consumers. To resolve these performance limitations, the team transitioned to storing segments as Roaring Bitmaps saved as single blobs in object storage. This compression strategy splits 32-bit integer user IDs into chunks across array, bitmap, and run containers based on data density, reducing a one-million-member segment to under one megabyte. An accompanying client SDK manages segment retrieval, decoding, update notifications, and least-recently-used in-memory caching. Consequently, consumers such as Grab's communications platform achieved peak throughput of 15,000 queries per second with sub-millisecond p99 read latencies. ### [Go module proxy at Grab](https://yomu.fyi/post/go-module-proxy-at-grab.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Jerry Ng - Published: Jun 30, 2023 Grab's 69.3 GiB multi-module Go monorepo caused commands like go get to take over 18 minutes as Git repeatedly traversed commit history, downloaded large worktrees, and overloaded their GitLab VCS infrastructure. To bypass direct VCS queries without losing automatic updates for external repositories, the team deployed the Athens Go module proxy configured in fallback network mode. They used the GOVCS environment variable to disable Git access specifically for the monorepo path, forcing Athens to fall back to its internal object storage when resolving monorepo modules. A dedicated CI pipeline pre-populates and refreshes the Athens cache whenever new monorepo modules are released. This setup reduced monorepo go get execution times to approximately 12 seconds and allowed a 70% scale-down of the Athens proxy cluster. ### [Performance bottlenecks of Go application on Kubernetes with non-integer (floating) CPU allocation](https://yomu.fyi/post/performance-bottlenecks-of-go-application-on-kubernetes-with-non-integ.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Shubham Badkur - Published: May 23, 2023 Grab's real-time stream processing platform encountered severe consumer lag and CPU throttling when running Go-based Kafka consumer pipelines on Kubernetes. The issue originated when the Vertical Pod Autoscaler (VPA) scaled pod CPU allocations down to floating-point values such as 1.94 cores. Because AUTO-GOMAXPROCS rounds non-integer CPU limits down to integers, Go runtime thread allocation dropped to 1 core, significantly throttling pipeline throughput despite available pod capacity. Setting a minimum floor of 2 cores instantly restored CPU utilization to 95% and cleared the message backlog. To prevent similar throttling, the team utilized integer CPU scaling recommendations available in VPA v0.13 on Kubernetes 1.25 and above. ### [How we improved our iOS CI infrastructure with observability tools](https://yomu.fyi/post/how-we-improved-our-ios-ci-infrastructure-with-observability-tools.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Bunty Madan - Published: May 18, 2023 Upgrading to Xcode 13.1 introduced severe CI test instability and high CPU utilisation for Grab's iOS development team. To address this, the team integrated observability tools across their UITest pipeline to pinpoint performance bottlenecks and test flakiness. Interventions included isolating spotlight.app to curb CPU spikes, replacing Safari with a mock browser for deep link tests, and booting simulators with pre-granted permissions. The team also built custom network tracking tools to enforce resource mocking and transitioned tests away from arbitrary sleep commands to explicit wait wrappers. These combined changes cut CI runtimes, decreased CPU utilisation by over 50%, and stabilized automated test executions. ### [2.3x faster using the Go plugin to replace Lua virtual machine](https://yomu.fyi/post/2-3x-faster-using-the-go-plugin-to-replace-lua-virtual-machine.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Yonghao Hu - Published: May 15, 2023 Talaria, an open-source distributed time-series database developed at Grab, previously allowed users to run custom data transformation scripts during ingestion using a Lua virtual machine. Launching and executing Lua scripts caused significant performance overhead when processing large volumes of events. To resolve this bottleneck, the team replaced the Lua VM with Go plugins compiled as Linux shared libraries (.so files). Benchmarks revealed that calling Go plugins achieves performance on par with native Go functions, executing roughly 2.3 times faster and consuming 2.3 times less memory than cached Lua VMs. Both execution methods conform to a unified Handler interface to load and run custom transformations. ### [How KartaCam powers GrabMaps](https://yomu.fyi/post/how-kartacam-powers-grabmaps.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Shuangquan Hou - Published: Dec 1, 2022 Grab's Geo team required an efficient, low-cost way to collect fresh street-level imagery across Southeast Asia, where professional mapping equipment is prohibitively expensive and smartphone crowdsourcing yields inconsistent quality. To address this, Grab built KartaCam, a custom mapping device equipped with a 12MP sensor, dual-band GNSS, 4G LTE, and onboard edge AI. Edge machine learning models evaluate scene suitability, check image quality, filter for map-relevant objects, and blur personal data directly on the device prior to upload. Deploying a four-camera KartaCam 360 array delivers panoramic coverage and point-of-interest data comparable to commercial mapping rigs at roughly one-twentieth the hardware cost. [Newer posts](https://yomu.fyi/topic/performance/page/2.md) · [Older posts](https://yomu.fyi/topic/performance/page/4.md)