How We Added Client Rate Throttling to the Platform API Gem

Summary

The platform-api gem needed client-side rate throttling for Heroku’s API, which uses GCRA and allows a bucket of up to 4,500 request tokens, replenished at roughly 75 per minute. The goal was to avoid 429 responses without creating retry traffic, while operating within the client library and without coordination across machines. The work combined tests with a simulated server, processes and threads, charting, and metrics for average retry rate, maximum sleep, request-count variance, and workload-clearing time. The winning strategy adjusted sleep downward using the remaining request count relative to the 4,500-request maximum; in simulation it achieved a 3.07% retry rate, 17.32-second maximum sleep, 78.44 standard-deviation request count, and 84.23 seconds to clear 4,500 requests. The change shipped behind a null strategy before becoming the default in version 3.0.0+, with feedback reporting no significant breaks or severe application dysfunction.

Takeaways

1. 1

   The selected strategy achieved a 3.07% average retry rate, 17.32-second maximum sleep, and 78.44 standard-deviation request count in simulation.

2. 2

   Decreasing sleep in proportion to remaining bucket capacity helps clients respond faster when the request bucket refills.

3. 3

   Rate throttling was introduced behind a null strategy before being enabled by default in platform-api version 3.0.0+.