Rate Limiting

2 posts about Rate Limiting. Every summary links to the original.

Grab · Feb 19, 2025
Grab AI Gateway: Connecting Grabbers to multiple GenAI providers
Grab AI Gateway is a central access layer connecting Grabbers’ applications to multiple GenAI providers, including OpenAI, Azure, AWS Bedrock, and Google VertexAI. Built as a set of reverse proxies, it centralizes provider authentication, authorization, rate limiting, auditing, and capacity management while presenting a unified OpenAI-style API; users can switch models by changing the model parameter. Exploration keys support short-lived staging experiments, while reviewed use cases can use longer-term service keys, and the gateway exposes embedding, image, audio, fine-tuning, search, and in-house open source models. Shared capacity, dynamic routing, monitoring, cost attribution, and usage records support governance and efficiency, although batch traffic can interfere with latency-sensitive services and provider-specific SDK edge cases require integration testing. More than 3,000 Grabbers have requested exploration keys, and the gateway powers applications including ride-safety audio analysis, content moderation, SQL generation, incident automation, and Slack support.
Bjorn Jee
Ramp · Aug 11, 2022
Rate limiting with Redis
Rate limiting at Ramp must handle third-party API quotas, globally paced Celery tasks, and inbound API traffic limited by authenticated application identities rather than only mutable IP addresses. Celery’s per-worker controls were insufficient for global limits, while queue prefetching could let rate-limited tasks block workers and stall queues. The team created a general framework for comparing algorithms and evaluated fixed-window, sliding-window, leaky/token-bucket, and Generic Cell Rate Algorithm (GCRA) approaches. They selected GCRA for implementation simplicity and compute and memory efficiency, storing a theoretical arrival time in Redis and using a Redis lock to prevent races. The implementation also uses floating-point timestamps, Redis server time, key expiration, unit tests, and manual testing; it has been introduced gradually across the three use cases, with monitoring, while additional algorithms remain planned.
Kwuang Tang

Grab AI Gateway: Connecting Grabbers to multiple GenAI providers

Grab AI Gateway is a central access layer connecting Grabbers’ applications to multiple GenAI providers, including OpenAI, Azure, AWS Bedrock, and Google VertexAI. Built as a set of reverse proxies, it centralizes provider authentication, authorization, rate limiting, auditing, and capacity management while presenting a unified OpenAI-style API; users can switch models by changing the model parameter. Exploration keys support short-lived staging experiments, while reviewed use cases can use longer-term service keys, and the gateway exposes embedding, image, audio, fine-tuning, search, and in-house open source models. Shared capacity, dynamic routing, monitoring, cost attribution, and usage records support governance and efficiency, although batch traffic can interfere with latency-sensitive services and provider-specific SDK edge cases require integration testing. More than 3,000 Grabbers have requested exploration keys, and the gateway powers applications including ride-safety audio analysis, content moderation, SQL generation, incident automation, and Slack support.

Bjorn Jee

Rate limiting with Redis

Rate limiting at Ramp must handle third-party API quotas, globally paced Celery tasks, and inbound API traffic limited by authenticated application identities rather than only mutable IP addresses. Celery’s per-worker controls were insufficient for global limits, while queue prefetching could let rate-limited tasks block workers and stall queues. The team created a general framework for comparing algorithms and evaluated fixed-window, sliding-window, leaky/token-bucket, and Generic Cell Rate Algorithm (GCRA) approaches. They selected GCRA for implementation simplicity and compute and memory efficiency, storing a theoretical arrival time in Redis and using a Redis lock to prevent races. The implementation also uses floating-point timestamps, Redis server time, key expiration, unit tests, and manual testing; it has been introduced gradually across the three use cases, with monitoring, while additional algorithms remain planned.

Kwuang Tang