---
title: "Lakebase branching for CI/CD: lessons from 100 PR databases"
description: "Snorkel AI needed each pull request in its CI pipeline to have an isolated copy of production data for realistic testing, but restoring a full database copy took about 20 minutes per environment. They adopted Databricks’ Lakebase Postgres, whose copy-on-write branching creates writable branches in seconds regardless of database size because no data is copied at branch creation. AWS DMS continuously replicates the production Aurora cluster into Lakebase’s main branch, keeping it within about two minutes of the source. For every PR, the CI pipeline creates a branch via API, runs tests, and either deletes the branch or relies on automatic TTL expiration to clean up. Provisioning time dropped from over 30 minutes to under 60 seconds at p95, supporting about 100 concurrent PR branches on a 2.5 TB dataset with storage growing only for changes written."
---

# Lakebase branching for CI/CD: lessons from 100 PR databases

[Databricks](https://yomu.fyi/company/databricks) · Yasamin Tari, Kartik Mathur · Nov 1, 2026

**Type:** Problem & solution

## Summary

Snorkel AI needed each pull request in its CI pipeline to have an isolated copy of production data for realistic testing, but restoring a full database copy took about 20 minutes per environment. They adopted Databricks’ Lakebase Postgres, whose copy-on-write branching creates writable branches in seconds regardless of database size because no data is copied at branch creation. AWS DMS continuously replicates the production Aurora cluster into Lakebase’s main branch, keeping it within about two minutes of the source. For every PR, the CI pipeline creates a branch via API, runs tests, and either deletes the branch or relies on automatic TTL expiration to clean up. Provisioning time dropped from over 30 minutes to under 60 seconds at p95, supporting about 100 concurrent PR branches on a 2.5 TB dataset with storage growing only for changes written.

## Context

Snorkel AI needed to provide isolated production-like databases for each pull request in its CI/CD pipeline without long restore times.

## Approach / What changed

Use Lakebase copy-on-write branching: replicate production Aurora to Lakebase main branch via DMS; create a branch per PR via API with TTL; authenticate developers with Okta SSO and CI pipelines with Databricks service principals and short-lived OAuth tokens; delete branch or rely on TTL for cleanup.

## Takeaways

- Copy-on-write branching creates writable database branches in seconds regardless of base size because no data is copied; only changed pages consume new storage.
- Setting a TTL on every branch is required to avoid long-term storage growth from branches retaining older page versions after main updates.
- CI pipelines authenticate as Databricks service principals using short-lived OAuth tokens via the Databricks SDK, eliminating long-lived passwords; all queries are audited by identity.

## FAQ

### How does Lakebase ensure data isolation between branches?

Each branch uses copy-on-write at the storage level; changes are written only to pages of that branch, so branches are fully isolated from each other and the parent. Compute is also per branch.

### What authentication does Lakebase support for CI pipelines?

CI pipelines use Databricks service principals provisioned via SCIM. They get short-lived OAuth tokens through the Databricks SDK; no long-lived passwords are stored in CI environments.

### How is the main branch kept close to production?

AWS DMS continuously replicates from production Aurora into Lakebase's main branch, first a full load then ongoing change data capture. It keeps the main branch within about two minutes of production, including large LOB columns.

**Tags:** [CI/CD](https://yomu.fyi/topic/ci-cd), [Copy-on-Write](https://yomu.fyi/topic/copy-on-write), [Database Branching](https://yomu.fyi/topic/database-branching), [Lakebase](https://yomu.fyi/topic/lakebase), [Postgres](https://yomu.fyi/topic/postgres)

- Source: [Databricks](https://www.databricks.com/blog/lakebase-branching-cicd-lessons-100-pr-databases)
- Source URL: https://www.databricks.com/blog/lakebase-branching-cicd-lessons-100-pr-databases
- Ingested by Yomu: 2026-11-01T00:00:15.144Z

[Read original post](https://www.databricks.com/blog/lakebase-branching-cicd-lessons-100-pr-databases)
