---
title: "How I Broke `git push heroku main`"
description: "Incident #2105 began during a production deployment of a Go-based Git server after a configuration variable had been set but not transmitted to production instances. Staging deployment completed successfully before the same change was attempted in production. The deployment sent a new code version to one batch, whose unauthenticated calls to a private API produced HTTP 401 responses that were returned publicly; the author paused before deploying the remaining batches and resolved the issue by synchronizing configuration, killing affected processes, and allowing them to restart. The post argues that the failure was enabled by a process requiring separate, infrequent manual actions rather than being solely an individual omission. Its proposed remedy is to combine configuration synchronization with code deployment, or alternatively warn when deployment begins with unsynchronized variables, supported by a blameless retrospective."
---

# How I Broke \`git push heroku main\`

[Heroku](https://yomu.fyi/company/heroku) · Damien Mathieu · Oct 1, 2020

**Type:** Incident / postmortem

## Summary

Incident #2105 began during a production deployment of a Go-based Git server after a configuration variable had been set but not transmitted to production instances. Staging deployment completed successfully before the same change was attempted in production. The deployment sent a new code version to one batch, whose unauthenticated calls to a private API produced HTTP 401 responses that were returned publicly; the author paused before deploying the remaining batches and resolved the issue by synchronizing configuration, killing affected processes, and allowing them to restart. The post argues that the failure was enabled by a process requiring separate, infrequent manual actions rather than being solely an individual omission. Its proposed remedy is to combine configuration synchronization with code deployment, or alternatively warn when deployment begins with unsynchronized variables, supported by a blameless retrospective.

## Context

A production deployment required separate manual actions to set a configuration variable, transmit it to instances, and deploy code. The variable was set in production but not transmitted before deployment, allowing instances without the required configuration to make unauthenticated calls to a private API.

## Approach / What changed

The affected processes were killed so they could restart with synchronized configuration. The longer-term solution is to merge configuration updates and code deployment into one step; an alternative is a confirmation warning when variables are not synchronized.

## Takeaways

- The Git server runs as a kernel app on virtual servers because it must accept SSH connections, while the Heroku platform provides HTTP routing.
- A production batch deployed without the new configuration made unauthenticated private-API calls, causing HTTP 401 responses to be sent back publicly.
- The proposed process improvement is to remove the separate synchronization step or add a deployment safeguard, rather than assigning blame for a missed command.

**Tags:** [Automation](https://yomu.fyi/topic/automation), [Deployment](https://yomu.fyi/topic/deployment), [Heroku](https://yomu.fyi/topic/heroku), [Incident Response](https://yomu.fyi/topic/incident-response)

- Source: [Heroku](https://www.heroku.com/blog/how-i-broke-git-push-heroku-main)
- Source URL: https://www.heroku.com/blog/how-i-broke-git-push-heroku-main
- Ingested by Yomu: 2026-09-09T20:02:34.355Z

[Read original post](https://www.heroku.com/blog/how-i-broke-git-push-heroku-main)