# Performance
> 114 posts about Performance, summarised, each linking to the original.

## Articles

### [Counter Service: How we rewrote it in Rust](https://yomu.fyi/post/counter-service-how-we-rewrote-it-in-rust.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Jia Long Loh
- Published: Jun 20, 2025

The Integrity Data Platform team rewrote Counter Service, a high-throughput Golang microservice serving event counts for fraud rules and machine learning models, to evaluate the operational return on investment of Rust. Rather than performing a line-by-line translation, engineers approached the service as a black box, reimplementing core application logic from scratch to satisfy established gRPC contracts across Scylla and Redis. The team resolved internal Go tooling dependencies by building custom configuration template parsers using the nom parser combinator and selected targeted open-source crates such as fred.rs and Cadence. Adapting to Rust required navigating cooperative, stackless async execution compared to Go's preemptive concurrency model, alongside managing borrow checker constraints. Ultimately, the rewrite achieved a 70% reduction in infrastructure costs while maintaining comparable service performance.


### [How On-Device AI Models Find Your Best Tinder Profile Photos](https://yomu.fyi/post/how-on-device-ai-models-find-your-best-tinder-profile-photos.md)
- Company: [Tinder](https://yomu.fyi/company/tinder.md)
- Author: Tinder
- Published: Apr 28, 2025

Tinder developed an on-device AI Photo Selector to recommend compelling profile photos directly from a user's camera roll without exposing private assets. To establish a reference face, the system extracts facial landmarks via Apple's Vision framework and normalizes the crop using affine transformations. To prevent funnel drop-offs from mandatory selfies, the architecture includes a fallback mechanism that detects a single recognizable face in existing profile photos. Processing workflows execute concurrently across thousands of local assets while downloading and initializing TensorFlow Lite models in parallel using Combine. A dedicated analytics queue synchronizes operation counts with DispatchGroup to ensure accurate metric capture before the global rollout.


### [Evaluating performance impact of removing Redis-cache from a Scylla-backed service](https://yomu.fyi/post/evaluating-performance-impact-of-removing-redis-cache-from-a-scylla-ba.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Md Riyadh
- Published: Apr 11, 2025

Grab operates a high-throughput Rust read service that aggregates counter metrics from Scylla tables across minutely, hourly, and daily granularities. The service initially cached aggregated responses in Redis using keys rounded to 15-minute intervals alongside a five-minute TTL. Because incoming queries predominantly requested recent data, transitions between 15-minute windows caused simultaneous cache misses across active configurations, resulting in severe Scylla traffic spikes, latency surges, and timeouts. To resolve the load imbalance, engineers proposed removing the Redis cache entirely and relying directly on Scylla's native internal caching. The rollout was staged in production by deterministically disabling Redis caching for specific counter configurations using mathematical operations on configuration IDs.


### [Scaling Nextdoor’s Datastores: Part 2](https://yomu.fyi/post/scaling-nextdoor-s-datastores-part-2.md)
- Company: [Nextdoor](https://yomu.fyi/company/nextdoor.md)
- Author: Tushar Singla
- Published: Mar 19, 2025

Nextdoor encountered scaling issues after adding read replicas when product engineers were initially tasked with deciding whether to route queries to the primary or replica databases. As business logic grew and gained abstraction layers, engineers struggled to track read-after-write consistency constraints across the call stack. To avoid replication lag race conditions, engineers routinely wrapped logic in database transactions, unintentionally directing all queries to the primary node and eroding read replica benefits over several years. The Core-Services team resolved this by injecting custom tracking logic into their Django ORM layer to monitor table writes during web requests and automate routing. They further optimized the system using a timing-based approach that restored replica read eligibility after the p99.9 replication lag elapsed.


### [How the Tinder iOS App reduced the size of our localizations by 95% using Emerge](https://yomu.fyi/post/how-the-tinder-ios-app-reduced-the-size-of-our-localizations-by-95-usi.md)
- Company: [Tinder](https://yomu.fyi/company/tinder.md)
- Author: Tinder
- Published: Feb 26, 2025

Supporting over 50 languages across numerous statically linked targets led to a substantial build size footprint in Tinder's iOS application. Because Apple's code signing mandates a minimum 4KB per file, shipping dozens of localized files per target inflated the final application package. Tinder addressed this by stripping comments and whitespace, merging localized strings into a single file per language via custom Bazel rules and Aspects, and compressing the strings using Emerge's SmallStrings tool into LZFSE files. The runtime decompresses these files dynamically through existing code-generated string accessors. Consequently, Tinder reduced download size by 10.7MB and install size by 51.3MB with no impact on developer workflows.


### [Sharing Tinder’s latest contributions to the open source community](https://yomu.fyi/post/sharing-tinder-s-latest-contributions-to-the-open-source-community.md)
- Company: [Tinder](https://yomu.fyi/company/tinder.md)
- Author: Tinder
- Published: Jan 29, 2025

Tinder open-sourced several iOS development repositories, including Layout, Nodes Architecture Framework, and CombineUI, to share the engineering patterns supporting its iPhone application. As the app expanded across diverse device profiles and scale, engineering teams faced reliability, consistency, and memory challenges under their legacy architecture. In response, Tinder developed Nodes, a plugin-based architecture framework using compile-time dependency injection and lifecycle hooks that enforce complete memory release upon feature dismissal. For interface construction, Tinder created Layout, a domain-specific Auto Layout wrapper offering declarative syntax for UIKit views to eliminate storyboard merge conflicts while preserving native capabilities. The resulting stack enables isolated testing of business logic, native reactive event binding, and incremental adoption of SwiftUI across the codebase.


### [The science of routing print orders](https://yomu.fyi/post/the-science-of-routing-print-orders.md)
- Company: [Canva](https://yomu.fyi/company/canva.md)
- Author: Constantinos Kavadias
- Published: Dec 10, 2024

Canva's global print network requires selecting optimal suppliers to balance delivery times, packaging counts, and environmental emissions. To resolve these challenges before user checkout, the engineering team designed a modular routing architecture that decouples graph construction, decision logic, and path traversal. During graph traversal, the system generates action objects capturing forward paths and decision query results, which are compiled into timestamped routing logs in blob storage for asynchronous auditing. Utilizing preprocessed graph queries alongside ElastiCache, Redis, and database read replicas, the infrastructure sustains high-throughput evaluation without coupling cost logic to traversal code. As a result, print routing completes within an average of 50 milliseconds at the 99th percentile during peak usage while maintaining 99.999% data availability.


### [How we reduced initialisation time of Product Configuration Management SDK](https://yomu.fyi/post/how-we-reduced-initialisation-time-of-product-configuration-management.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Ram Dilip Pradhan
- Published: Nov 22, 2024

GrabX operates as Grab's central platform for product configuration management, where client services fetch configuration data via an eventually consistent SDK. Services handling around 400 MB of configuration data experienced startup cold starts taking approximately four minutes, creating service stress during traffic spikes. The engineering team resolved this bottleneck through a multi-phase optimization of how the SDK retrieves data from AWS S3. First, sequential downloads of common and service-specific datasets were replaced with concurrent fetching. Next, concurrent downloading and memory loading were applied across large configurations within subscribed services, followed by the complete removal of an outdated disk-caching fallback mechanism. Benchmarks across diverse configuration payloads showed an overall initialisation time reduction of up to 90%.


### [How we reduced peak memory and CPU usage of the product configuration management SDK](https://yomu.fyi/post/how-we-reduced-peak-memory-and-cpu-usage-of-the-product-configuration.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Ram Dilip Pradhan
- Published: Oct 30, 2024

Grab's central product configuration management platform, GrabX, previously aggregated all configurations across every backend service into a single JSON file hosted on AWS S3. Every minute, client SDKs fetched, parsed, and loaded this growing file—which exceeded 100MB—causing CPU throttling spikes, elevated P99 latency, and unnecessary memory consumption. Analysis revealed that 98% of services required less than 1% of the total configuration data. To resolve these bottlenecks, the team partitioned data by service, split configurations into separate S3 files under distinct prefixes, and introduced a per-service changelog for incremental updates. Benchmarks showed the redesign decreased maximum CPU utilisation by over 50% and reduced memory usage by up to 70%.


### [Improving Compute Sustainability: A Case Study](https://yomu.fyi/post/improving-compute-sustainability-a-case-study.md)
- Company: [Two Sigma](https://yomu.fyi/company/two-sigma.md)
- Author: Emily Majewski
- Published: Oct 7, 2024

Two Sigma's large computing footprint drives significant energy consumption and carbon emissions, particularly across live trading applications that require continuous real-time market data caching. To address this overhead without sacrificing performance, engineering teams used a routine hardware refresh to transition from legacy single-process machines to denser multi-core server configurations. By replacing roughly 60 legacy hosts with 28-core processor hardware, the team distributed baseline power draw over more cores and eliminated underutilized compute capacity. This architectural consolidation reduced absolute power consumption across production hosts by 66%, dropping electricity usage from 27 MWh in January 2023 to 10 MWh in January 2024. The initiative subsequently established an annual sustainability rationalization practice for hardware budgeting across the organization.


### [Profile-guided optimisation (PGO) on Grab services](https://yomu.fyi/post/profile-guided-optimisation-pgo-on-grab-services.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Yonghao Hu
- Published: Jun 5, 2024

Profile-guided optimisation (PGO) improves Go application performance by feeding production CPU profile data into subsequent compiler builds. Teams evaluated PGO across multiple Grab services, including TalariaDB and an orchestrator within the Catwalk service, using Go 1.20 and Go 1.21 builds. TalariaDB saw substantial resource savings after compiling Docker images with a 360-second pprof profile, resulting in at least a 10% drop in CPU usage, a 30% drop in memory consumption, and a 38% reduction in persistent volume usage for event ingestion queues. Conversely, testing on Catwalk showed only a 5% performance gain after adjusting the profiling duration from 59 seconds to 6 minutes, an outcome deemed insufficient to justify pipeline modifications. Broader adoption across monorepo services remains constrained until the central build process supports PGO arguments and automated profile retrieval.


### [Iris - Turning observations into actionable insights for enhanced decision making](https://yomu.fyi/post/iris-turning-observations-into-actionable-insights-for-enhanced-decisi.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Huong Vuong
- Published: Apr 3, 2024

Standard infrastructure monitoring tools track CPU and memory usage at the host or container level, obscuring the precise resource consumption of individual Spark jobs on shared instances. To achieve granular observability, the Iris platform gathers job metrics directly at the Java Virtual Machine level without requiring changes to user application code. Data collection relies on Uber's JVM Profiler for deep JVM, thread, and memory pool diagnostics alongside sparkMeasure for stage and task execution statistics. Collected metrics route asynchronously through an Apache Kafka queue to avoid execution bottlenecks, feeding a Telegraf, InfluxDB, and Grafana stack for real-time monitoring while archiving to a data lake for offline analytics. This multi-tiered telemetry enables engineering teams to identify over- or under-provisioned jobs and optimize ETL resource allocations.


### [NeurIPS 2023: Our Favorite Papers on LLMs, Statistical Learning, and More](https://yomu.fyi/post/neurips-2023-our-favorite-papers-on-llms-statistical-learning-and-more.md)
- Company: [Two Sigma](https://yomu.fyi/company/two-sigma.md)
- Author: Emily Majewski
- Published: Mar 21, 2024

Researchers reviewed prominent machine learning papers presented at NeurIPS 2023 covering large language models and statistical learning theory. One investigation showed that claimed emergent abilities in models such as GPT-3 often result from nonlinear evaluation metrics rather than fundamental shifts in model capability. To reduce the computational burden of model adaptation, QLoRA enables 65-billion-parameter model fine-tuning on a single 48-gigabyte GPU via 4-bit NormalFloat quantization and paged optimization. Direct Preference Optimization eliminates complex reward modeling by casting reinforcement learning from human feedback into a preference classification task. Additional work resolved statistical anomalies like double descent using effective parameter counts and introduced stochastic gradient approximations for Gaussian processes.


### [Android App Size at Scale with Project Bonsai](https://yomu.fyi/post/android-app-size-at-scale-with-project-bonsai.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Nguyen Van Minh
- Published: Mar 1, 2024

Grab launched Project Bonsai to optimize the size of its Android superapp, which encompassed over four million lines of code across 1,500 modules. Because large app sizes negatively impact user acquisition on low-end devices with constrained bandwidth, the engineering team structured its optimization strategy around measurement, reduction, and containment. The team developed App Sizer, a custom CI-integrated tool that tracks binary compositions and module size contributions in Grafana. Initial reductions targeted Java and Kotlin dex files, resolving bloated transitive R classes by upgrading the Android Gradle Plugin to eliminate overly broad R8 retention rules. These initiatives achieved a 26% reduction in app download size while simultaneously decreasing overall disk footprint.


### [Enabling near real-time data analytics on the data lake](https://yomu.fyi/post/enabling-near-real-time-data-analytics-on-the-data-lake.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Shi Kai Ng
- Published: Feb 23, 2024

Traditional data lake setups using Parquet on Hive metastores struggle with frequent updates and long pipeline intervals, introducing significant latency for ad hoc queries. Grab solved this by implementing Apache Hudi to support near real-time analytics across bounded relational databases and unbounded Kafka streams. For high-throughput sources, Flink streams Avro log files to Merge On Read tables and generates compaction plans for asynchronous Spark writers. Low-throughput workloads leverage Copy On Write tables, while relational database sources ingest binlogs via Flink Change Data Capture connectors. This architecture reduced analytics data latency to the minute level without overloading production databases and Kafka clusters.


### [The journey of building a comprehensive attribution platform](https://yomu.fyi/post/the-journey-of-building-a-comprehensive-attribution-platform.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Kang Huang
- Published: Feb 20, 2024

Grab needed to modernize its marketing analytics from manual ad hoc queries and high data latency to a platform supporting real-time attribution for pricing models like cost per order. The engineering team initially deployed a pure stream-processing engine using Kappa architecture, Kafka, ScyllaDB, and Redis, which reduced latency from days to minutes and merged ads and promo touchpoints. However, stream-only processing faced high costs, out-of-order event issues, and difficulties running multi-touch models across longer historical windows. Grab transitioned to a Lambda architecture pairing Coban stream processing with Spark-based batch ETL and Amazon S3. This hybrid design separated real-time operational metrics from historical batch reporting, cutting real-time processing costs by approximately 25% while maintaining under 1% data discrepancy.


### [Sliding window rate limits in distributed systems](https://yomu.fyi/post/sliding-window-rate-limits-in-distributed-systems.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Naveen Kumar Jakuva Premkumar
- Published: Dec 14, 2023

Marketing communications across Grab's user base risked causing notification overload and consent revocations. To enforce personalized daily and weekly frequency caps across more than 270 million users, the team addressed segment membership storage and communication rate limiting. They adopted roaring bitmaps instead of Bloom filters to compactly store user segment data without hash collisions or costly rebuilds upon deletion. For frequency capping, they chose Amazon ElastiCache for Redis over DynamoDB, executing a sliding log rate limiting algorithm directly on the cluster using Lua scripts and sorted sets. Timestamps are stored as sorted set scores, and historical data is cleaned up via eviction ranges to prevent unbounded memory growth.


### [Road localisation in GrabMaps](https://yomu.fyi/post/road-localisation-in-grabmaps.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Roxana Crisan
- Published: Nov 17, 2023

Grab needed to localise nearly 30 million road segments across more than 10,000 area boundaries in Southeast Asia to attach hyperlocal attributes like language, driving side, and vehicle access rules. Testing direct geometric inclusion between complex road polylines and boundary polygons proved computationally prohibitive for daily map generation pipelines. To optimize this process, Grab introduced geohashes as rectangular proxies, precomputing geohash coverage across borders and road segments before joining the datasets in parallel. To resolve misclassification errors near borders without sacrificing performance, the team evaluated geohash coverage percentages and refined boundary-touching geohashes with polygon clipping. A laptop benchmark on a subset of data reduced runtime from 38 minutes with geometric intersection to 78 seconds using the geohash approximation.


### [Stepping up marketing for advertisers: Scalable lookalike audience](https://yomu.fyi/post/stepping-up-marketing-for-advertisers-scalable-lookalike-audience.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: William Wu
- Published: Sep 22, 2023

Grab's legacy lookalike audience generation platform suffered from long creation SLAs of two working days, high costs, and low weekly update frequencies. To resolve these bottlenecks, the engineering team designed an embedding-based platform powered by an in-memory retrieval service and automated update pipelines. The system creates audience representations by averaging constituent passenger embeddings, determining user membership through real-time cosine score thresholds. To eliminate feature store latency while fitting all embeddings into memory, a hash-based compression method cuts passenger embedding storage needs by roughly 90%. Consequently, audience availability dropped to within 15 minutes of campaign creation, audience generation costs fell by 98%, and ad impressions and clicks doubled.


### [Streamlining Grab's Segmentation Platform with faster creation and lower latency](https://yomu.fyi/post/streamlining-grab-s-segmentation-platform-with-faster-creation-and-low.md)
- Company: [Grab](https://yomu.fyi/company/grab.md)
- Author: Jake Ng
- Published: Aug 15, 2023

Grab's Segmentation Platform previously stored user-to-segment mappings across individual rows in ScyllaDB, causing write bottlenecks during segment creation and read latencies too high for downstream consumers. To resolve these performance limitations, the team transitioned to storing segments as Roaring Bitmaps saved as single blobs in object storage. This compression strategy splits 32-bit integer user IDs into chunks across array, bitmap, and run containers based on data density, reducing a one-million-member segment to under one megabyte. An accompanying client SDK manages segment retrieval, decoding, update notifications, and least-recently-used in-memory caching. Consequently, consumers such as Grab's communications platform achieved peak throughput of 15,000 queries per second with sub-millisecond p99 read latencies.


[Newer posts](https://yomu.fyi/topic/performance/page/3.md) · [Older posts](https://yomu.fyi/topic/performance/page/5.md)