How I Broke `git push heroku main`

Summary

Incident #2105 began during a production deployment of a Go-based Git server after a configuration variable had been set but not transmitted to production instances. Staging deployment completed successfully before the same change was attempted in production. The deployment sent a new code version to one batch, whose unauthenticated calls to a private API produced HTTP 401 responses that were returned publicly; the author paused before deploying the remaining batches and resolved the issue by synchronizing configuration, killing affected processes, and allowing them to restart. The post argues that the failure was enabled by a process requiring separate, infrequent manual actions rather than being solely an individual omission. Its proposed remedy is to combine configuration synchronization with code deployment, or alternatively warn when deployment begins with unsynchronized variables, supported by a blameless retrospective.

Takeaways

1. 1

   The Git server runs as a kernel app on virtual servers because it must accept SSH connections, while the Heroku platform provides HTTP routing.

2. 2

   A production batch deployed without the new configuration made unauthenticated private-API calls, causing HTTP 401 responses to be sent back publicly.

3. 3

   The proposed process improvement is to remove the separate synchronization step or add a deployment safeguard, rather than assigning blame for a missed command.