Loading…
Topic
8 posts about Rate Limiting. Every summary links to the original.
See the idea
A model deployed on AKS is open by default. See how MaaS policies insert Authorino authentication and Limitador rate limits in front of vLLM, so bad keys and over-quota requests stop at the gate.
The API gateway is the single boundary where protocol translation, auth, quotas, and routing become shared policy instead of per-service code. This layered diagram shows what belongs there and why.
Backpressure is a feedback loop that slows work when downstream capacity or health deteriorates. Compare its gains, costs, and signals for choosing a control point.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Red Hat AI Inference 3.5 extends its Model-as-a-Service (MaaS) governance layer to non-OpenShift Kubernetes distributions, including Azure Kubernetes Service (AKS). MaaS adds multi-tenant API governance on top of the inference platform, controlling authentication, rate limiting, and model access via Authorino and Limitador from Red Hat Connectivity Link (Kuadrant). A model is first deployed with a single LLMInferenceService YAML, creating a vLLM server and router-scheduler, but the endpoint is open. Applying a MaaSSubscription and MaaSAuthPolicy activates governance: Authorino validates API keys or JWTs, and Limitador enforces token rate limits. The post demonstrates three enforcement scenarios on an AKS cluster with a T4 GPU running Qwen3-0.6B: 401 for missing or invalid keys, 200 for authenticated requests, and 429 when rate limits are exceeded, blocking requests before they reach vLLM.
Salesforce’s Cloud Atlas team operates a globally distributed identity data store that must maintain five nines of availability, roughly five minutes of downtime per year. The team redesigned service protection after per-instance rate limiting proved insufficient under autoscaling and increasingly bursty, unpredictable agent-driven traffic. The new approach combines intelligent load shedding, which monitors queue time as an early indicator of system pressure before error rates spike, with coordinator-free global quota management. Servers independently measure and share customer usage to throttle noisy tenants without a centralized coordinator, preventing one customer’s surge from affecting others. This design prevents cascading failures by shedding lower-priority work before overload spreads across dependent services, ensuring graceful degradation rather than widespread outage.
SMS pumping fraud, also called SMS toll fraud, uses bots and fake phone numbers to trigger automated text messages from public-facing endpoints, leaving businesses to pay for traffic that produces no genuine customers. The attack can target OTP requests, sign-up and lead forms, transactional notifications, marketing campaigns, and chatbot endpoints; it may appear as a sudden spike or as distributed low-volume traffic that accumulates over days or weeks. Suggested detection measures include monitoring traffic by use case and geography, reconciling spend with conversions, examining clustered phone-number patterns, and analyzing incomplete forms. The prevention guidance covers CAPTCHA, behavioral bot detection, rate limits, geographic controls, delayed OTP requests, alternative verification methods, and Twilio products including Verify Fraud Guard, SMS Pumping Protection, and Lookup SMS Pumping Risk Score.
Auth0 429 responses in production are tied to rate limits, commonly when applications call the Management API for every request or obtain a new machine-to-machine (M2M) token for every request. Because limits are per tenant and vary by plan, logging x-ratelimit-limit, x-ratelimit-remaining, and x-ratelimit-reset headers exposes approaching exhaustion. User roles, permissions, and metadata that can be treated as access-token snapshots should be added as custom claims by a Post-Login Action, while APIs read them from the JWT. M2M tokens, valid for 24 hours by default, should be cached and refreshed with a 60-second expiry buffer; data requiring freshness should sit behind a server-side cache with a 30-to-60-second TTL. When a 429 occurs, retry at the Unix timestamp in x-ratelimit-reset rather than applying exponential backoff, because the reset time is deterministic.
Grab AI Gateway is a central access layer connecting Grabbers’ applications to multiple GenAI providers, including OpenAI, Azure, AWS Bedrock, and Google VertexAI. Built as a set of reverse proxies, it centralizes provider authentication, authorization, rate limiting, auditing, and capacity management while presenting a unified OpenAI-style API; users can switch models by changing the model parameter. Exploration keys support short-lived staging experiments, while reviewed use cases can use longer-term service keys, and the gateway exposes embedding, image, audio, fine-tuning, search, and in-house open source models. Shared capacity, dynamic routing, monitoring, cost attribution, and usage records support governance and efficiency, although batch traffic can interfere with latency-sensitive services and provider-specific SDK edge cases require integration testing. More than 3,000 Grabbers have requested exploration keys, and the gateway powers applications including ride-safety audio analysis, content moderation, SQL generation, incident automation, and Slack support.
Rate limiting at Ramp must handle third-party API quotas, globally paced Celery tasks, and inbound API traffic limited by authenticated application identities rather than only mutable IP addresses. Celery’s per-worker controls were insufficient for global limits, while queue prefetching could let rate-limited tasks block workers and stall queues. The team created a general framework for comparing algorithms and evaluated fixed-window, sliding-window, leaky/token-bucket, and Generic Cell Rate Algorithm (GCRA) approaches. They selected GCRA for implementation simplicity and compute and memory efficiency, storing a theoretical arrival time in Redis and using a Redis lock to prevent races. The implementation also uses floating-point timestamps, Redis server time, key expiration, unit tests, and manual testing; it has been introduced gradually across the three use cases, with monitoring, while additional algorithms remain planned.
The platform-api gem needed client-side rate throttling for Heroku’s API, which uses GCRA and allows a bucket of up to 4,500 request tokens, replenished at roughly 75 per minute. The goal was to avoid 429 responses without creating retry traffic, while operating within the client library and without coordination across machines. The work combined tests with a simulated server, processes and threads, charting, and metrics for average retry rate, maximum sleep, request-count variance, and workload-clearing time. The winning strategy adjusted sleep downward using the remaining request count relative to the 4,500-request maximum; in simulation it achieved a 3.07% retry rate, 17.32-second maximum sleep, 78.44 standard-deviation request count, and 84.23 seconds to clear 4,500 requests. The change shipped behind a null strategy before becoming the default in version 3.0.0+, with feedback reporting no significant breaks or severe application dysfunction.
The post, adapted from Joe Kutner’s Devoxx 2018 talk, presents ten ways to strengthen web application security, with examples focused on Java but applicable to modern languages. It recommends keeping dependencies current, validating explicit payloads with schemas, checking regular expressions for ReDoS, throttling abusive requests, and keeping credentials outside the codebase. It also advises secure-first coding, certificate validation for outgoing connections, logging security-relevant actions without recording secrets, and monitoring suspicious behavior with tools such as AppSensor. The examples include Maven Versions Plugin, Snyk, Bucket4j, saferegex, environment variables, and EnvKeyStore, with a token-bucket algorithm proposed for request throttling. The conclusion is that security requires anticipating unintended functionality across network, platform, and physical domains, while these practices provide a starting point.
The best engineering reads, once a week.
