# Performance > 120 posts about Performance, summarised, each linking to the original. ## Articles ### [Streamlining Grab's Segmentation Platform with faster creation and lower latency](https://yomu.fyi/post/streamlining-grab-s-segmentation-platform-with-faster-creation-and-low.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Jake Ng - Published: Aug 15, 2023 Grab's Segmentation Platform previously stored user-to-segment mappings across individual rows in ScyllaDB, causing write bottlenecks during segment creation and read latencies too high for downstream consumers. To resolve these performance limitations, the team transitioned to storing segments as Roaring Bitmaps saved as single blobs in object storage. This compression strategy splits 32-bit integer user IDs into chunks across array, bitmap, and run containers based on data density, reducing a one-million-member segment to under one megabyte. An accompanying client SDK manages segment retrieval, decoding, update notifications, and least-recently-used in-memory caching. Consequently, consumers such as Grab's communications platform achieved peak throughput of 15,000 queries per second with sub-millisecond p99 read latencies. ### [Go module proxy at Grab](https://yomu.fyi/post/go-module-proxy-at-grab.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Jerry Ng - Published: Jun 30, 2023 Grab's 69.3 GiB multi-module Go monorepo caused commands like go get to take over 18 minutes as Git repeatedly traversed commit history, downloaded large worktrees, and overloaded their GitLab VCS infrastructure. To bypass direct VCS queries without losing automatic updates for external repositories, the team deployed the Athens Go module proxy configured in fallback network mode. They used the GOVCS environment variable to disable Git access specifically for the monorepo path, forcing Athens to fall back to its internal object storage when resolving monorepo modules. A dedicated CI pipeline pre-populates and refreshes the Athens cache whenever new monorepo modules are released. This setup reduced monorepo go get execution times to approximately 12 seconds and allowed a 70% scale-down of the Athens proxy cluster. ### [Performance bottlenecks of Go application on Kubernetes with non-integer (floating) CPU allocation](https://yomu.fyi/post/performance-bottlenecks-of-go-application-on-kubernetes-with-non-integ.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Shubham Badkur - Published: May 23, 2023 Grab's real-time stream processing platform encountered severe consumer lag and CPU throttling when running Go-based Kafka consumer pipelines on Kubernetes. The issue originated when the Vertical Pod Autoscaler (VPA) scaled pod CPU allocations down to floating-point values such as 1.94 cores. Because AUTO-GOMAXPROCS rounds non-integer CPU limits down to integers, Go runtime thread allocation dropped to 1 core, significantly throttling pipeline throughput despite available pod capacity. Setting a minimum floor of 2 cores instantly restored CPU utilization to 95% and cleared the message backlog. To prevent similar throttling, the team utilized integer CPU scaling recommendations available in VPA v0.13 on Kubernetes 1.25 and above. ### [How we improved our iOS CI infrastructure with observability tools](https://yomu.fyi/post/how-we-improved-our-ios-ci-infrastructure-with-observability-tools.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Bunty Madan - Published: May 18, 2023 Upgrading to Xcode 13.1 introduced severe CI test instability and high CPU utilisation for Grab's iOS development team. To address this, the team integrated observability tools across their UITest pipeline to pinpoint performance bottlenecks and test flakiness. Interventions included isolating spotlight.app to curb CPU spikes, replacing Safari with a mock browser for deep link tests, and booting simulators with pre-granted permissions. The team also built custom network tracking tools to enforce resource mocking and transitioned tests away from arbitrary sleep commands to explicit wait wrappers. These combined changes cut CI runtimes, decreased CPU utilisation by over 50%, and stabilized automated test executions. ### [2.3x faster using the Go plugin to replace Lua virtual machine](https://yomu.fyi/post/2-3x-faster-using-the-go-plugin-to-replace-lua-virtual-machine.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Yonghao Hu - Published: May 15, 2023 Talaria, an open-source distributed time-series database developed at Grab, previously allowed users to run custom data transformation scripts during ingestion using a Lua virtual machine. Launching and executing Lua scripts caused significant performance overhead when processing large volumes of events. To resolve this bottleneck, the team replaced the Lua VM with Go plugins compiled as Linux shared libraries (.so files). Benchmarks revealed that calling Go plugins achieves performance on par with native Go functions, executing roughly 2.3 times faster and consuming 2.3 times less memory than cached Lua VMs. Both execution methods conform to a unified Handler interface to load and run custom transformations. ### [How KartaCam powers GrabMaps](https://yomu.fyi/post/how-kartacam-powers-grabmaps.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Shuangquan Hou - Published: Dec 1, 2022 Grab's Geo team required an efficient, low-cost way to collect fresh street-level imagery across Southeast Asia, where professional mapping equipment is prohibitively expensive and smartphone crowdsourcing yields inconsistent quality. To address this, Grab built KartaCam, a custom mapping device equipped with a 12MP sensor, dual-band GNSS, 4G LTE, and onboard edge AI. Edge machine learning models evaluate scene suitability, check image quality, filter for map-relevant objects, and blur personal data directly on the device prior to upload. Deploying a four-camera KartaCam 360 array delivers panoramic coverage and point-of-interest data comparable to commercial mapping rigs at roughly one-twentieth the hardware cost. ### [Typeahead Search at Nextdoor](https://yomu.fyi/post/typeahead-search-at-nextdoor.md) - Company: [Nextdoor](https://yomu.fyi/company/nextdoor.md) - Author: Jerry Tian - Published: Jul 6, 2022 Nextdoor built a proximity-based autocomplete service to power typeahead search and mention features across its hyperlocal platform for hundreds of millions of entities, including users and businesses. The system shards geographic data using Uber's open-source H3 geohashing library and stores prefix indexes in memory using Redis sorted sets. By adopting a Command Query Responsibility Segregation architecture, ingestion writes are processed on Redis primary nodes and replicated to read-only search nodes with under 10 milliseconds of replication lag. Dedicated APIs handle indexing, typeahead lookups, and ranking before returning hydrated results. Operating since August 2021, the service processes hundreds of millions of monthly typeahead queries while maintaining a P95 search latency below 30 milliseconds. ### [Supporting large campaigns at scale](https://yomu.fyi/post/supporting-large-campaigns-at-scale.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Jie Zhang - Published: Apr 1, 2022 Grab developed a batch job service within its Trident automation engine to execute multi-step marketing campaigns for millions of users simultaneously. The system replaces sequential, single-server execution with a distributed architecture powered by Apache Kafka, which distributes batches of 100 users across server clusters using hashed partition keys. To reduce network overhead and queries per second, downstream reward and messaging services introduced batch endpoints backed by bulk database queries, decreasing API latency by up to 85%. Grab further optimized performance by sharding Kafka topics by country and action type to prevent long-running reward tasks from blocking time-sensitive messaging workloads. Additionally, making terminal messaging calls asynchronous allows subsequent batch processing to proceed without waiting for message delivery confirmations. ### [Using real-world patterns to improve matching in theory and practice](https://yomu.fyi/post/using-real-world-patterns-to-improve-matching-in-theory-and-practice.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Tenindra Abeywickrama - Published: Nov 22, 2021 Continuous ride-hailing assignment relies on solving the minimum weight bipartite matching problem between passengers and driver-partners. While traditional implementations assume a precalculated cost matrix, computing shortest-path travel times across large road networks dominates total execution time. Researchers introduced an Incremental Kuhn-Munkres algorithm that leverages the spatial locality of optimal matches to compute edge costs on demand. The approach integrates priority queues and lower-bounding techniques with refinement rules to avoid evaluating distant pairs while guaranteeing the same optimal assignment. Evaluated on Singapore road network data and real Grab production workloads, the incremental techniques reduced exact cost calculations and decreased assignment running times by over an order of magnitude. ### [How We Cut GrabFood.com’s Page JavaScript Asset Sizes by 3x](https://yomu.fyi/post/how-we-cut-grabfood-com-s-page-javascript-asset-sizes-by-3x.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Gibson Cheng - Published: Jul 29, 2021 GrabFood.com faced high cloud costs while serving over 1 TB of weekly network egress across 175 million requests. To minimize egress and improve page performance, the engineering team audited and reduced their webpack dependencies using tools such as webpack-bundle-analyzer and dependency-cruiser. Their strategies included lazy loading rarely used libraries, unifying duplicate modules under single entry points, and substituting utility libraries like axios with native browser Web APIs. Additionally, altering implementation approaches, such as replacing client-side signed JWT cookie encoding with plain JSON strings, eliminated heavy Node dependencies like crypto. These optimizations reduced JavaScript static assets from 750 KB to 250 KB, decreased CloudFront costs by 20%, and accelerated build times by 3.6x. ### [Supabase Community Day](https://yomu.fyi/post/supabase-community-day.md) - Company: [Supabase](https://yomu.fyi/company/supabase.md) - Author: Steve Chavez - Published: Jul 26, 2021 Supabase announced a series of platform upgrades and open-source contributions centered around PostgreSQL and its broader tooling ecosystem. Every new project on the platform now deploys on PostgreSQL version 13.3, accompanied by marketplace images on Digital Ocean and AWS. PostgREST version 8.0 incorporates upstreamed performance enhancements, achieving up to 50% more throughput on GET requests by adopting prepared statements for SELECT queries and decreasing log verbosity. The release eliminates downtime during schema cache reloads and replaces the pg\_listen requirement with standard NOTIFY commands for dynamic in-database configuration updates. Additionally, community-led efforts introduced beta releases of Flutter and Dart client libraries alongside a dedicated Discord server for developer discussions. ### [Supabase is now on Postgres 13.3](https://yomu.fyi/post/supabase-is-now-on-postgres-13-3.md) - Company: [Supabase](https://yomu.fyi/company/supabase.md) - Author: Angelico de los Reyes - Published: Jul 26, 2021 New Supabase projects now run on Supabase Postgres 13.3, transitioning the database platform forward from PostgreSQL 12.0. This release introduces features such as native UUID generation with gen\_random\_uuid, parallel index vacuuming, incremental sorting, smaller btree indexes, and extended statistics improvements. The underlying PostgreSQL binary was built from the ground up to enable Large System Extensions on AWS Graviton2 ARM instances, optimizing locking and synchronization performance. Alongside upgrading the underlying operating system to Ubuntu 20.04 for long-term support through 2025, the team integrated fail2ban to protect direct database connections on ports 5432 and 6543 against brute-force attacks. Finally, Supabase Postgres version numbering now mirrors upstream PostgreSQL major and minor versions to make tracking installed releases easier. ### [App Modularisation at Scale](https://yomu.fyi/post/app-modularisation-at-scale.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Amar Jain - Published: Jul 13, 2021 Grab transitioned its monolithic mobile application into a modular architecture to resolve increasing code conflicts, slow releases, and difficult team collaboration. The team decomposed the single module by establishing base infrastructure modules, shared UI and utility libraries, discrete feature modules, and bridge kit modules for inter-module communication. Dependency injection using Dagger ties these components together in the main app module while preventing feature modules from directly depending on one another. The architecture spans over 1,000 modules across the app, with more than 200 modules in the Grab Financial Group payments domain where over 95% of modules build in under 15 seconds. This approach accelerated Gradle CI and local builds through parallel compilation and caching, though it increased Gradle sync times, IDE memory usage, and configuration maintenance overhead. ### [Debugging High Latency Due to Context Leaks](https://yomu.fyi/post/debugging-high-latency-due-to-context-leaks.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Sourabh Suman - Published: Jun 30, 2021 Market-Store, Grab's feature store for real-time machine learning features, experienced latency spikes from under 200 milliseconds to 2 seconds as traffic grew. Metrics and logs showed no direct correlation to API issues, but heap profiling with PPROF revealed continuously increasing memory held by child contexts. Further analysis tracked the leak to an update in Grab's open-source Async Library, which switched background contexts to uncancelled task contexts for worker runners. Because parent contexts maintained references to these uncancelled child contexts, the garbage collector could not reclaim their memory. This progressive memory exhaustion directly degraded API latency. ### [How We Improved Agent Chat Efficiency with Machine Learning](https://yomu.fyi/post/how-we-improved-agent-chat-efficiency-with-machine-learning.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Suman Anand - Published: Apr 19, 2021 Agent typing time represented a large portion of Grab's chat support journey, and 85% of messages were still free typed because agents customized static templates to fit their personal style. To accelerate typing across multilingual markets without robotic templates, Grab built SmartChat, a machine learning feature that provides contextual sentence completion. The team opted for a lightweight seq2seq architecture using single-layered GRU encoder-decoders in TensorFlow instead of bulky attention models to keep model latency under 100ms. The user interface was implemented in React using a content-editable div with inline typeahead suggestions activated via keyboard shortcuts. ### [How Grab Leveraged Performance Marketing Automation to Improve Conversion Rates by 30%](https://yomu.fyi/post/how-grab-leveraged-performance-marketing-automation-to-improve-convers.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Sc Ng - Published: Mar 22, 2021 Grab faced operational bottlenecks managing direct-response Google Ads campaigns across thousands of ad groups due to its hyperlocal marketing across Southeast Asian markets. To eliminate the manual burden of tracking and updating ad creatives, the team built CARA, an in-house automation tool deployed on AWS serverless compute. CARA utilizes standardized file naming conventions to map assets to specific campaigns and connects with Google Ads and YouTube APIs to detect and replace low-performing assets. During an experimental rollout across more than 8,000 active ad groups, CARA replaced nearly 2,000 underperforming creatives. The automated asset replacement workflow produced an 18% to 30% increase in clickthrough and conversion rates. ### [Supabase Beta January 2021](https://yomu.fyi/post/supabase-beta-january-2021.md) - Company: [Supabase](https://yomu.fyi/company/supabase.md) - Author: Paul Copplestone - Published: Feb 2, 2021 Supabase released a collection of platform updates during January 2021 covering database tools, authentication integrations, regional availability, and infrastructure performance. Client libraries added support for PostgREST exact, planned, and estimated count queries to eliminate counting friction commonly experienced in platforms like Firebase. Authentication capabilities expanded to include Facebook and Azure login providers, an in-dashboard GoTrue audit trail, a helper function for user emails, and a prebuilt React Auth Widget. The platform also enabled new database hosting regions in London and Sydney while transitioning storage to GP3 for improved throughput. Furthermore, migrating API subdomains to Route53 alongside custom Let's Encrypt certificates yielded read performance benchmark gains of up to twelve percent. ### [Serving Driver-partners Data at Scale Using Mirror Cache](https://yomu.fyi/post/serving-driver-partners-data-at-scale-using-mirror-cache.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Indrajit Sarkar - Published: Jan 26, 2021 Grab's Drivers Data service handles up to 10,000 requests per second during peak hours to supply driver information across backend microservices. The original setup used MySQL with Redis and standalone in-memory local caches, but yielded a low 25% local cache hit rate due to traffic patterns characterized by high burst frequency for individual drivers alongside redundant database calls across nodes. To solve this, the team developed Mirror Cache, an in-memory caching system that pairs Dgraph's Ristretto library with an asynchronous gRPC replication layer to mirror updates across cluster nodes. The replicator batches updates within the same AWS availability zone and forwards data to single nodes across zones to minimize transfer overhead. Production deployment increased the in-memory cache hit rate to approximately 75% and reduced direct MySQL queries by 5%. ### [Trident - Real-time Event Processing at Scale](https://yomu.fyi/post/trident-real-time-event-processing-at-scale.md) - Company: [Grab](https://yomu.fyi/company/grab.md) - Author: Jie Zhang - Published: Jan 13, 2021 Trident serves as Grab's internal real-time event-processing and workflow automation engine, driving user campaigns, rewards, and notifications across multiple business lines. To handle peak loads exceeding 2,000 events per second without duplicate execution, the system consumes decoupled Kafka streams and enforces exactly-once semantics using Redis and MySQL deduplication checks. Processing efficiency relies on server autoscaling aligned with Kafka partition counts, combined with dynamic goroutine allocation per consumer. To minimize rule evaluation overhead, Trident indexes active campaigns into an in-memory hash map by event type, cutting processing time by at least 90%. Furthermore, condition evaluation is optimized through lazy loading and a weighted sorting algorithm that checks low-cost in-memory data prior to executing expensive database queries or external service calls. ### [Supabase Beta December 2020](https://yomu.fyi/post/supabase-beta-december-2020.md) - Company: [Supabase](https://yomu.fyi/company/supabase.md) - Author: Paul Copplestone - Published: Jan 2, 2021 Supabase transitioned to Beta following ten months of development concentrated on performance, security, and reliability. The platform introduced inline table documentation updates, storing descriptions directly as PostgreSQL comments and exposing them through OpenAPI specifications. Developers can now utilize a table view featuring immediate realtime update reflection alongside pagination for handling large datasets. Performance measurements show a 26 percent increase to 1460 requests per second in regions supporting AWS Graviton instances. Furthermore, the update introduces a new region in Sao Paulo, editing support for native Postgres arrays in the grid editor and side panel, and broader compatibility for custom database types. [Newer posts](https://yomu.fyi/topic/performance/page/4.md) · [Older posts](https://yomu.fyi/topic/performance/page/6.md)