GithubLawrence GripperProblem & solution
How GitHub uses eBPF to improve deployment safety
Summary
Deployment scripts can introduce dangerous circular dependencies when they rely on services or assets from platforms that are currently experiencing outages. Blocking network access at the host level is impractical because stateful nodes continue serving live traffic during rolling deployments. To solve this, GitHub isolates deploy scripts into dedicated Linux cGroups and attaches custom eBPF programs via the cilium/ebpf Go library. The system uses socket-address hooks to redirect DNS queries to a userspace proxy that checks a domain blocklist, while egress packet hooks map DNS transaction IDs to process IDs. This approach successfully prevents deploy-time circular dependencies, provides full command-line audit logs for blocked requests, and speeds up incident recovery.
Context
Deployment scripts executing on stateful production nodes risked failing during incidents due to circular dependencies on GitHub, but completely blocking host-level network access would break live customer traffic.
Approach / What changed
GitHub isolated deployment scripts within Linux cGroups and applied eBPF programs (BPF_PROG_TYPE_CGROUP_SOCK_ADDR and BPF_PROG_TYPE_CGROUP_SKB) alongside a userspace DNS proxy to intercept, filter, and audit outbound domain requests per process.
Takeaways
Using BPF_PROG_TYPE_CGROUP_SOCK_ADDR enables rewriting connect4 syscalls on port 53 to route cGroup DNS queries to a local userspace proxy.
Correlating skb_buff DNS transaction IDs with process IDs in eBPF maps allows looking up /proc/{PID}/cmdline to identify the exact command triggering a blocked request.
Placing deploy scripts in dedicated cGroups allows applying network egress blocklists without impacting the host's primary production traffic.
Source and freshness
- Source
- github.blog
- Published
- Added to Yomu
- Last updated
Summaries are generated from the original by a language model. Check the original before relying on it.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
The best engineering reads, once a week
One useful email. Unsubscribe anytime.