Incident Preparation: Simulating Third-Party Downtime

Summary

Heroku’s help application depends on the Platform API for authentication, authorization, and user data, so it must remain available to agents and customers during platform incidents. The open-sourced api-maintenance-sim app tests this requirement by imitating an outage in which every request receives an HTTP 503 Service Unavailable response. After deploying the simulator, teams point the application under test to its hostname through PLATFORM_API_URL, passed to platform-api’s connect_oauth method; when unset, the gem uses the production URI. A heroku config:set command enables the alternate endpoint, making regular simulations possible and helping reveal what failure means for the application or service when the Platform API is unavailable, supporting preparation for incidents and communication during outages.

Takeaways

1. 1

   api-maintenance-sim imitates a Platform API incident by returning a 503 Service Unavailable response for every request.

2. 2

   The platform-api gem accepts a custom endpoint through PLATFORM_API_URL; when the variable is unset, it uses the production URI.

3. 3

   Regular downtime simulations can reveal how an application or service behaves when the Platform API is unavailable.