Enabling Evolutionary Database Development: database branching with Lakebase, continued

Summary

This installment revisits Evolutionary Database Design and argues that Databricks Lakebase removes the infrastructure constraints that kept several of its database-change practices aspirational. Lakebase is a managed Postgres database with compute separated from shared durable storage, while copy-on-write branches create a new pointer and divergence marker in roughly one second without copying parent data. That enables per-developer, per-PR, and per-experiment databases, real Postgres test branches instead of mocks or in-memory substitutes, and CI isolation at pull-request granularity. The updated playbook adds idempotent migrations, destructive testing, and database-level A/B prototyping, with Unity Catalog governance inherited by branches and agents receiving branches rather than production access. Jen’s example corrupts production-shaped data to test an inventory-code migration and compares column and lookup-table designs, rejecting the latter because its common read path requires a join.

Context

Evolutionary Database Design practices were limited by the cost and operational burden of provisioning isolated database instances. Shared databases constrained DBA collaboration, prevented per-pipeline isolation, made database refactoring harder to rehearse at PR granularity, and made on-demand experimentation unsafe.

Approach / What changed

Use Databricks Lakebase’s managed Postgres architecture and copy-on-write branching to create isolated, production-shaped database branches for developers, pull requests, experiments, tests, and agents. The revised playbook incorporates schema diffs, idempotent migrations, destructive testing, parallel design branches, and inherited Unity Catalog governance.

Takeaways

• Lakebase branches are metadata operations that share unchanged storage with their parent and diverge only on modified pages, completing in roughly one second regardless of parent size.
• Per-branch Postgres environments replace mocks, H2, SQLite, and shared staging for database-change testing, while enabling destructive tests with a zero-cost reset.
• Parallel database branches let teams compare schema designs against production-shaped data; Jen rejected a lookup table because its common inventory read path required a join.

Related reading