# No version left behind: Our epic journey of GitLab upgrades

[Grab](https://yomu.fyi/company/grab) · Saurabh Vajpayee · May 3, 2024

**Type:** Problem & solution

## Summary

Grab's self-hosted GitLab instance fell approximately 14 months behind official releases, serving thousands of engineers and monorepos reaching up to 39TB in total footprint. To address security updates and system stability, the team established a structured upgrade routine using Terraform, Packer, and Ansible across GitLab's 5,000-user reference architecture. They implemented staged deployments, upgraded stateful Gitaly nodes via in-place rotation, and resolved primary node clustering bottlenecks under Praefect. By utilizing official GitLab Upgrade Paths and parallelizing component deployments, the team caught up on 24 months of releases in 11 months and reduced release lag from 396 days down to 35 days.

## Context

Grab's self-hosted GitLab deployment trailed official releases by roughly 14 months due to stability issues and learning curve challenges, risking security vulnerabilities across a platform supporting thousands of engineers and large monorepos.

## Approach / What changed

The team established a comprehensive preparation checklist, refreshed staging with production data, utilized GitLab Upgrade Paths to skip versions, deployed immutable AMIs via Terraform with parallel stateless component updates, and resolved Gitaly node sync distribution issues.

## Takeaways

- Using official GitLab Upgrade Paths enabled Grab to catch up on 24 months of updates across an 11-month period rather than upgrading through every minor version sequentially.
- Upgrading stateful Gitaly nodes via in-place rotation can cause Praefect to concentrate monorepos onto single overloaded primary nodes, requiring explicit primary node validation during deployment.
- Separating Git protocols—encouraging HTTPS links for CI and automation scripts while keeping SSH for developer workflows—mitigated RPC concurrency bottlenecks and rate limiting.

**Tags:** [Architecture](https://yomu.fyi/topic/architecture), [CI/CD](https://yomu.fyi/topic/ci-cd), [Deployment](https://yomu.fyi/topic/deployment), [Migrations](https://yomu.fyi/topic/migration), [Reliability](https://yomu.fyi/topic/reliability)

[Read original post](https://engineering.grab.com/no-version-left-behind-our-epic-journey-of-gitlab-upgrades)