Loading…
Performance
114 posts about Performance. Every summary links to the original.
Grab ·
Go module proxy at Grab
Grab's 69.3 GiB multi-module Go monorepo caused commands like go get to take over 18 minutes as Git repeatedly traversed commit history, downloaded large worktrees, and overloaded their GitLab VCS infrastructure. To bypass direct VCS queries without losing automatic updates for external repositories, the team deployed the Athens Go module proxy configured in fallback network mode. They used the GOVCS environment variable to disable Git access specifically for the monorepo path, forcing Athens to fall back to its internal object storage when resolving monorepo modules. A dedicated CI pipeline pre-populates and refreshes the Athens cache whenever new monorepo modules are released. This setup reduced monorepo go get execution times to approximately 12 seconds and allowed a 70% scale-down of the Athens proxy cluster.
Jerry NgGrab ·
Performance bottlenecks of Go application on Kubernetes with non-integer (floating) CPU allocation
Grab's real-time stream processing platform encountered severe consumer lag and CPU throttling when running Go-based Kafka consumer pipelines on Kubernetes. The issue originated when the Vertical Pod Autoscaler (VPA) scaled pod CPU allocations down to floating-point values such as 1.94 cores. Because AUTO-GOMAXPROCS rounds non-integer CPU limits down to integers, Go runtime thread allocation dropped to 1 core, significantly throttling pipeline throughput despite available pod capacity. Setting a minimum floor of 2 cores instantly restored CPU utilization to 95% and cleared the message backlog. To prevent similar throttling, the team utilized integer CPU scaling recommendations available in VPA v0.13 on Kubernetes 1.25 and above.
Shubham BadkurGrab ·
How we improved our iOS CI infrastructure with observability tools
Upgrading to Xcode 13.1 introduced severe CI test instability and high CPU utilisation for Grab's iOS development team. To address this, the team integrated observability tools across their UITest pipeline to pinpoint performance bottlenecks and test flakiness. Interventions included isolating spotlight.app to curb CPU spikes, replacing Safari with a mock browser for deep link tests, and booting simulators with pre-granted permissions. The team also built custom network tracking tools to enforce resource mocking and transitioned tests away from arbitrary sleep commands to explicit wait wrappers. These combined changes cut CI runtimes, decreased CPU utilisation by over 50%, and stabilized automated test executions.
Bunty MadanGrab ·
2.3x faster using the Go plugin to replace Lua virtual machine
Talaria, an open-source distributed time-series database developed at Grab, previously allowed users to run custom data transformation scripts during ingestion using a Lua virtual machine. Launching and executing Lua scripts caused significant performance overhead when processing large volumes of events. To resolve this bottleneck, the team replaced the Lua VM with Go plugins compiled as Linux shared libraries (.so files). Benchmarks revealed that calling Go plugins achieves performance on par with native Go functions, executing roughly 2.3 times faster and consuming 2.3 times less memory than cached Lua VMs. Both execution methods conform to a unified Handler interface to load and run custom transformations.
Yonghao HuGrab ·
How KartaCam powers GrabMaps
Grab's Geo team required an efficient, low-cost way to collect fresh street-level imagery across Southeast Asia, where professional mapping equipment is prohibitively expensive and smartphone crowdsourcing yields inconsistent quality. To address this, Grab built KartaCam, a custom mapping device equipped with a 12MP sensor, dual-band GNSS, 4G LTE, and onboard edge AI. Edge machine learning models evaluate scene suitability, check image quality, filter for map-relevant objects, and blur personal data directly on the device prior to upload. Deploying a four-camera KartaCam 360 array delivers panoramic coverage and point-of-interest data comparable to commercial mapping rigs at roughly one-twentieth the hardware cost.
Shuangquan HouNextdoor ·
Typeahead Search at Nextdoor
Nextdoor built a proximity-based autocomplete service to power typeahead search and mention features across its hyperlocal platform for hundreds of millions of entities, including users and businesses. The system shards geographic data using Uber's open-source H3 geohashing library and stores prefix indexes in memory using Redis sorted sets. By adopting a Command Query Responsibility Segregation architecture, ingestion writes are processed on Redis primary nodes and replicated to read-only search nodes with under 10 milliseconds of replication lag. Dedicated APIs handle indexing, typeahead lookups, and ranking before returning hydrated results. Operating since August 2021, the service processes hundreds of millions of monthly typeahead queries while maintaining a P95 search latency below 30 milliseconds.
Jerry TianGrab ·
Supporting large campaigns at scale
Grab developed a batch job service within its Trident automation engine to execute multi-step marketing campaigns for millions of users simultaneously. The system replaces sequential, single-server execution with a distributed architecture powered by Apache Kafka, which distributes batches of 100 users across server clusters using hashed partition keys. To reduce network overhead and queries per second, downstream reward and messaging services introduced batch endpoints backed by bulk database queries, decreasing API latency by up to 85%. Grab further optimized performance by sharding Kafka topics by country and action type to prevent long-running reward tasks from blocking time-sensitive messaging workloads. Additionally, making terminal messaging calls asynchronous allows subsequent batch processing to proceed without waiting for message delivery confirmations.
Jie ZhangGrab ·
Using real-world patterns to improve matching in theory and practice
Continuous ride-hailing assignment relies on solving the minimum weight bipartite matching problem between passengers and driver-partners. While traditional implementations assume a precalculated cost matrix, computing shortest-path travel times across large road networks dominates total execution time. Researchers introduced an Incremental Kuhn-Munkres algorithm that leverages the spatial locality of optimal matches to compute edge costs on demand. The approach integrates priority queues and lower-bounding techniques with refinement rules to avoid evaluating distant pairs while guaranteeing the same optimal assignment. Evaluated on Singapore road network data and real Grab production workloads, the incremental techniques reduced exact cost calculations and decreased assignment running times by over an order of magnitude.
Tenindra AbeywickramaGrab ·
How We Cut GrabFood.com’s Page JavaScript Asset Sizes by 3x
GrabFood.com faced high cloud costs while serving over 1 TB of weekly network egress across 175 million requests. To minimize egress and improve page performance, the engineering team audited and reduced their webpack dependencies using tools such as webpack-bundle-analyzer and dependency-cruiser. Their strategies included lazy loading rarely used libraries, unifying duplicate modules under single entry points, and substituting utility libraries like axios with native browser Web APIs. Additionally, altering implementation approaches, such as replacing client-side signed JWT cookie encoding with plain JSON strings, eliminated heavy Node dependencies like crypto. These optimizations reduced JavaScript static assets from 750 KB to 250 KB, decreased CloudFront costs by 20%, and accelerated build times by 3.6x.
Gibson ChengGrab ·
App Modularisation at Scale
Grab transitioned its monolithic mobile application into a modular architecture to resolve increasing code conflicts, slow releases, and difficult team collaboration. The team decomposed the single module by establishing base infrastructure modules, shared UI and utility libraries, discrete feature modules, and bridge kit modules for inter-module communication. Dependency injection using Dagger ties these components together in the main app module while preventing feature modules from directly depending on one another. The architecture spans over 1,000 modules across the app, with more than 200 modules in the Grab Financial Group payments domain where over 95% of modules build in under 15 seconds. This approach accelerated Gradle CI and local builds through parallel compilation and caching, though it increased Gradle sync times, IDE memory usage, and configuration maintenance overhead.
Amar JainGrab ·
Debugging High Latency Due to Context Leaks
Market-Store, Grab's feature store for real-time machine learning features, experienced latency spikes from under 200 milliseconds to 2 seconds as traffic grew. Metrics and logs showed no direct correlation to API issues, but heap profiling with PPROF revealed continuously increasing memory held by child contexts. Further analysis tracked the leak to an update in Grab's open-source Async Library, which switched background contexts to uncancelled task contexts for worker runners. Because parent contexts maintained references to these uncancelled child contexts, the garbage collector could not reclaim their memory. This progressive memory exhaustion directly degraded API latency.
Sourabh SumanGrab ·
How We Improved Agent Chat Efficiency with Machine Learning
Agent typing time represented a large portion of Grab's chat support journey, and 85% of messages were still free typed because agents customized static templates to fit their personal style. To accelerate typing across multilingual markets without robotic templates, Grab built SmartChat, a machine learning feature that provides contextual sentence completion. The team opted for a lightweight seq2seq architecture using single-layered GRU encoder-decoders in TensorFlow instead of bulky attention models to keep model latency under 100ms. The user interface was implemented in React using a content-editable div with inline typeahead suggestions activated via keyboard shortcuts.
Suman AnandGrab ·
How Grab Leveraged Performance Marketing Automation to Improve Conversion Rates by 30%
Grab faced operational bottlenecks managing direct-response Google Ads campaigns across thousands of ad groups due to its hyperlocal marketing across Southeast Asian markets. To eliminate the manual burden of tracking and updating ad creatives, the team built CARA, an in-house automation tool deployed on AWS serverless compute. CARA utilizes standardized file naming conventions to map assets to specific campaigns and connects with Google Ads and YouTube APIs to detect and replace low-performing assets. During an experimental rollout across more than 8,000 active ad groups, CARA replaced nearly 2,000 underperforming creatives. The automated asset replacement workflow produced an 18% to 30% increase in clickthrough and conversion rates.
Sc NgGrab ·
Serving Driver-partners Data at Scale Using Mirror Cache
Grab's Drivers Data service handles up to 10,000 requests per second during peak hours to supply driver information across backend microservices. The original setup used MySQL with Redis and standalone in-memory local caches, but yielded a low 25% local cache hit rate due to traffic patterns characterized by high burst frequency for individual drivers alongside redundant database calls across nodes. To solve this, the team developed Mirror Cache, an in-memory caching system that pairs Dgraph's Ristretto library with an asynchronous gRPC replication layer to mirror updates across cluster nodes. The replicator batches updates within the same AWS availability zone and forwards data to single nodes across zones to minimize transfer overhead. Production deployment increased the in-memory cache hit rate to approximately 75% and reduced direct MySQL queries by 5%.
Indrajit SarkarGrab ·
Trident - Real-time Event Processing at Scale
Trident serves as Grab's internal real-time event-processing and workflow automation engine, driving user campaigns, rewards, and notifications across multiple business lines. To handle peak loads exceeding 2,000 events per second without duplicate execution, the system consumes decoupled Kafka streams and enforces exactly-once semantics using Redis and MySQL deduplication checks. Processing efficiency relies on server autoscaling aligned with Kafka partition counts, combined with dynamic goroutine allocation per consumer. To minimize rule evaluation overhead, Trident indexes active campaigns into an in-memory hash map by event type, cutting processing time by at least 90%. Furthermore, condition evaluation is optimized through lazy loading and a weighted sorting algorithm that checks low-cost in-memory data prior to executing expensive database queries or external service calls.
Jie ZhangSupabase ·
Postgres Views
Postgres views serve as query shortcuts that execute underlying SQL statements upon retrieval without generating new tables or persisting duplicate data. By encapsulating complex multi-table joins, standard views provide query consistency across applications, simplify repetitive calls, improve logical schema organization, and enhance security by restricting sensitive columns. In contrast, materialized views physically store query results on disk, dramatically reducing read latency for heavy queries spanning millions of rows. Because materialized views introduce the trade-off of stale data, administrators must periodically run the refresh command based on workload tolerances for use cases like analytics and internal dashboards. Materialized views should not substitute query optimization, as underlying query efficiency remains essential.
Paul CopplestoneGrab ·
Optimally Scaling Kafka Consumer Applications
Grab's Coban platform runs Golang-based stream processing pipelines on Kubernetes, servicing roughly 400 billion events weekly from Kafka. The initial Horizontal Pod Autoscaler setup caused resource waste and uneven load distribution across Kafka partitions during scale-in and scale-out events. To resolve this, Grab moved to a fixed pod count matching the topic's partition count and adopted Vertical Pod Autoscaling, reducing resource usage versus requests by approximately 45%. The team also introduced Kubernetes priority classes to segment latency-sensitive workloads onto On-Demand nodes and non-critical jobs onto Spot instances. Additionally, overprovisioning via low-priority placeholder pods managed by Cluster Proportional Autoscaler enabled rapid pod rescheduling and reduced deployment delays.
Shubham BadkurSupabase ·
Supabase Alpha September 2020
Seven months into development, Supabase announced a series of platform updates across authentication, database tooling, and client libraries. The release introduced OAuth logins supporting Bitbucket, GitHub, GitLab, and Google, alongside table cloning and one-click Postgres extension management. In the SQL editor, users can now save favorite queries and access locally stored query histories directly within the browser. The web dashboard adopted Next.js automatic static optimization for improved responsiveness, while postgrest-js migrated to TypeScript and an isomorphic gotrue-js TypeScript library was built for Netlify GoTrue integration. Supabase is prioritizing a transition from Alpha to Beta by tracking open-source tool performance in a dedicated benchmarks repository.
Paul CopplestoneGrab ·
Uncovering the Truth Behind Lua and Redis Data Consistency
Grab experienced replica CPU usage spikes following service deployments in their master/replica Redis cluster, which caused failovers to spike to 100% CPU. Investigation revealed that a post-deployment Lua monitor script executed separately on both nodes and relied on non-deterministic HGETALL key ordering. Redis encodes hash objects as either ziplists or hashtables, and restoring from an RDB snapshot initializes small hashes as ziplists even if the master previously converted them to hashtables. This encoding discrepancy caused key ordering to diverge, preventing secondary data from deleting correctly and bloating dataset sizes. Grab resolved the issue by sorting the outputs of HKEYS and HGETALL within the Lua script to guarantee deterministic execution across nodes.
Allen Wanghuggingface.co ·
The Reformer - Pushing the limits of language modeling
Standard transformer models hit memory bottlenecks on long sequence modeling tasks due to the quadratic asymptotic memory complexity of global self-attention and oversized positional embedding matrices. The Reformer architecture overcomes these constraints to train sequences of up to half a million tokens using under 8GB of RAM. It re-engineers transformer operations using local and Locality Sensitive Hashing self-attention, chunked feed forward layers, reversible residual layers, and axial positional encodings. In empirical benchmarks using google/reformer-crime-and-punishment, axial positional encodings reduce the model parameter count from over 136 million to approximately 2.58 million by factorizing the positional dimensions. This architectural change cuts inference memory consumption from 959 MB down to 447 MB for evaluated benchmark workloads.
Patrick von Platen