How We Designed the Quotas Microservice to Prevent Resource Abuse

Summary

As Grab migrated from a monolith to hundreds of microservices, managing global rate limiting became essential to prevent cascading failures and resource exhaustion. To avoid putting a rate limiting service on the critical path of every API call, Grab built Quotas, an asynchronous rate limiting system. Client services use a lightweight SDK and middleware to read rate limiting decisions from local in-memory caches and stream usage metrics asynchronously via Apache Kafka. The Quotas service aggregates usage data locally, flushes stats to Redis periodically, and publishes updated rate limiting decisions back over Kafka topics. In production, Quotas successfully handles 200k peak transactions per second with decision enforcement delays capped at 200 milliseconds.

Context

Migrating from a monolith to hundreds of microservices made local rate limiting insufficient, as auto-scaling instances could collectively overload databases and downstream services, while synchronous global rate limiting would introduce severe latency on the critical path.

Approach / What changed

Grab built Quotas, an asynchronous rate limiting architecture where client instances evaluate requests against local in-memory caches, report request events via Kafka, and consume updated decisions calculated by a centralized service that aggregates stats in Redis.

Takeaways

• Asynchronous event processing via Kafka decoupled rate limiting decisions from the synchronous request-response path, keeping client latency impacts negligible.
• Aggregating API usage stats locally in-memory and updating Redis periodically every 50 ms prevented the Redis cluster from becoming a calculation bottleneck.
• Quotas achieved global rate limiting with an enforcement decision delay of up to 200 milliseconds while supporting 200k peak production TPS using sliding windows.

Related reading

• Grab · Mar 20, 2019

  Designing Resilient Systems Beyond Retries (Part 1): Rate-Limiting

  Distributed systems that rely exclusively on retries and circuit breakers face severe failure risks, including retry storms and reliance on client-side configuration accuracy. Implementing server-side rate limiting serves as a critical defensive layer to safeguard services across evolving architectures. Throttling thresholds can be layered across per-client, per-endpoint, and server-wide granularities using algorithms such as leaky bucket or sliding windows. While local instance-level limits fail when downstream bottlenecks like databases saturate under horizontal scaling, global rate limiting coordinates traffic enforcement across entire service pools. Centralized rate limiters require asynchronous communication and fallback mechanisms to avoid becoming single points of failure or adding request path latency.

  Michael Cartmell

• Grab · Feb 18, 2022

  Exposing a Kafka Cluster via a VPC Endpoint Service

  To replace VPC peering and reduce attack surfaces, Grab exposed a multi-Availability Zone Apache Kafka cluster in its main AWS VPC to clients in a separate GrabKios VPC using AWS VPC Endpoint Service. Because Kafka requires clients to establish deterministic connections to individual brokers, the team configured a Network Load Balancer with unique TCP ports and dedicated target groups for each broker alongside a shared bootstrap port. They added custom listeners on the Kafka brokers to advertise endpoints using private Route 53 CNAMEs rather than raw interface hostnames. To eliminate unnecessary cross-AZ network latency and data transfer costs, the architecture was refined to advertise AZ-specific private CNAMEs mapped directly to zonal endpoint interfaces.

  Fabrice Harbulot

• Grab · Jul 16, 2025

  Grab's service mesh evolution: From Consul to Istio

  Grab operated over 1,000 microservices across hybrid infrastructure using Consul alongside a fallback mechanism called Catcher. Single-point-of-failure vulnerabilities in Consul servers and limited support for multi-cluster operations prompted an evaluation of alternative mesh technologies, ultimately leading to the selection of Istio. Grab avoided the standard single-control-plane-per-cluster pattern by deploying multiple external control planes in dedicated Kubernetes clusters arranged in active-active pairs. Migration began in Q4 2024, shifting traffic across AWS and GCP while handling both HTTP and gRPC protocols with gradual traffic-shifting and rollback mechanisms.

  Hilman Kurniawan

• Grab · Mar 3, 2019

  How We Simplified Our Data Ingestion & Transformation Process

  Grab evolved its real-time data ingestion pipeline after an initial architecture built on Spark Streaming and Python encountered operational complexity, node failures, and data loss from S3 eventual consistency. Because the streaming workload primarily handled event partitioning and ORC file generation, the team consolidated these tasks directly into an existing Golang processing service. They implemented sharded concurrent maps for high-throughput partitioning and optimized heap allocations to resolve memory bottlenecks. This refactor removed intermediate Avro conversions and intermediate storage hops. The simplified Go pipeline eliminated data loss and reduced processing lag from up to 13 minutes down to approximately 1 minute.

  Yichao Wang