Managing Google Cloud Platform Project-Wide SSH Keys

Summary

Google Cloud project-wide SSH keys can remain in project metadata indefinitely after a user connects to a VM, potentially allowing matching private-key holders to access Linux VMs with root access unless project-wide keys are blocked. Shopify developed SSH-Pruner because OS Login did not work with Google Kubernetes Engine, while deleting all keys also removed Google’s keys, disrupted scripts, and created slow metadata updates. The new Go application uses Google Cloud APIs to find organization projects with Compute Engine enabled, reads the “ssh-keys” metadata, parses entries, retains unexpired keys, writes pruned common instance metadata, and records readable logs; Google-managed “sshKeys” entries are skipped. SSH-Pruner was already running as a cron job on some projects, with a slow rollout planned to limit workflow disruption.

Context

User-created SSH public keys are stored in Google Cloud project-wide metadata and can provide access to Linux VMs across the project, potentially with root access. The keys persist indefinitely, while OS Login does not work with Google Kubernetes Engine. An earlier deletion approach also removed Google’s keys, disrupted scripts, and caused slow metadata updates.

Approach / What changed

Shopify created SSH-Pruner as a new Go application. It uses Google Cloud APIs to identify organization projects with Compute Engine enabled, reads project metadata, processes the “ssh-keys” entries line by line, retains unexpired keys, writes pruned common instance metadata, skips Google’s “sshKeys” entries, and produces readable logs.

Takeaways

• Project-wide SSH keys can remain usable across Linux VMs after their original connection unless project-wide public SSH keys are blocked.
• Deleting every project-wide key was unsuitable because it removed Google’s keys, could interrupt scripts, and made re-adding keys slow.
• SSH-Pruner distinguishes the “ssh-keys” and “sshKeys” metadata objects, retaining unexpired user keys while leaving Google-managed keys untouched.

Related reading