Snorkel AI needed each pull request in its CI pipeline to have an isolated copy of production data for realistic testing, but restoring a full database copy took about 20 minutes per environment. They adopted Databricks’ Lakebase Postgres, whose copy-on-write branching creates writable branches in seconds regardless of database size because no data is copied at branch creation. AWS DMS continuously replicates the production Aurora cluster into Lakebase’s main branch, keeping it within about two minutes of the source. For every PR, the CI pipeline creates a branch via API, runs tests, and either deletes the branch or relies on automatic TTL expiration to clean up. Provisioning time dropped from over 30 minutes to under 60 seconds at p95, supporting about 100 concurrent PR branches on a 2.5 TB dataset with storage growing only for changes written.
Context
Snorkel AI needed to provide isolated production-like databases for each pull request in its CI/CD pipeline without long restore times.
Approach / What changed
Use Lakebase copy-on-write branching: replicate production Aurora to Lakebase main branch via DMS; create a branch per PR via API with TTL; authenticate developers with Okta SSO and CI pipelines with Databricks service principals and short-lived OAuth tokens; delete branch or rely on TTL for cleanup.
Takeaways
1
Copy-on-write branching creates writable database branches in seconds regardless of base size because no data is copied; only changed pages consume new storage.
2
Setting a TTL on every branch is required to avoid long-term storage growth from branches retaining older page versions after main updates.
3
CI pipelines authenticate as Databricks service principals using short-lived OAuth tokens via the Databricks SDK, eliminating long-lived passwords; all queries are audited by identity.
FAQ
How does Lakebase ensure data isolation between branches?
Each branch uses copy-on-write at the storage level; changes are written only to pages of that branch, so branches are fully isolated from each other and the parent. Compute is also per branch.
What authentication does Lakebase support for CI pipelines?
CI pipelines use Databricks service principals provisioned via SCIM. They get short-lived OAuth tokens through the Databricks SDK; no long-lived passwords are stored in CI environments.
How is the main branch kept close to production?
AWS DMS continuously replicates from production Aurora into Lakebase's main branch, first a full load then ongoing change data capture. It keeps the main branch within about two minutes of production, including large LOB columns.
Checkpoint/Restore captures live state so work resumes instead of restarting. This architecture view maps the state surface, the capture mechanism, the store tier, and the restore path.
A tiered KV cache flow that moves blocks from GPU HBM to DRAM, local NVMe, and CephFS—and shows how to decide when restoring a block beats recomputing it.
chDB embeds a ClickHouse OLAP engine inside your process, so analytical queries skip the network round trip. This architecture view maps the binding layer, the local MergeTree working copy, and the flush path to object storage.
A trade-off map for change management: reversible changes can move fast with light gates, while irreversible ones need staged rollout, auditability, and adoption work.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.