Loading post…
DatabricksBenjamin Nwokeleme, Firas Farah, Jen DarrouzetTutorial
A practical guide to cost optimization with Lakebase Postgres
Summary
Lakebase uses separated storage and compute to reduce costs: branches share storage, autoscaling matches compute to demand, and scale to zero suspends compute after inactivity. Read replicas and high availability add compute without duplicating storage. For synced tables from Lakehouse, sync only the active working set via materialized views and choose the appropriate sync mode—snapshot for volatile data, triggered for incremental, continuous for real-time—and binpack tables into one pipeline to share compute. Right-size initial compute by focusing on the working set, and set PITR retention and snapshot schedules based on actual recovery needs. Snapshot storage is priced at $0.090/GB-month and PITR storage at $0.200/GB-month, both cheaper than regular branch storage. Compute and storage usage can be tracked through system.billing.usage, with separate SKUs for compute, branch storage, PITR, and snapshots.
Context
Lakebase is a fully managed Postgres database with separated storage and compute, tightly integrated with the Databricks Lakehouse platform.
Approach / What changed
Optimize Lakebase costs by: using branching to share storage across environments; configuring autoscaling with scale to zero for non-latency-sensitive workloads; syncing only the active working set from Lakehouse via materialized views and selecting the appropriate sync mode (snapshot, triggered, continuous); binpacking multiple tables into one sync pipeline; right-sizing compute based on working set size rather than total database; and managing PITR retention and snapshot schedules based on actual recovery needs. Monitor all usage via system.billing.usage.
Takeaways
Branching avoids duplicate storage by allowing environments like dev and test to share the same underlying storage, only tracking divergences from the parent branch.
For Synced Tables, sync only the active working set via materialized views and choose the sync mode based on data volatility: Snapshot for >10% changes, Triggered for incremental, Continuous for real-time.
Right-size compute by focusing on the working set size (frequently accessed data and indexes) rather than total database size, and adjust PITR and snapshot retention to match actual recovery needs rather than keeping excessive history.
FAQ
How does branching reduce storage costs?
Lakebase branches share the same underlying storage and track changes as the branch diverges, so creating a branch does not require a separate physical copy. This makes it cost-efficient for short-lived dev and test environments where teams need production-like data without provisioning a full duplicate.
Which sync mode is most cost-effective for volatile data?
Snapshot mode is recommended when more than 10% of source data changes per cycle, as it can be up to 10x more cost-efficient than incremental modes in that scenario. For incremental workloads, Triggered mode offers a good balance of cost and latency.
How can I monitor Lakebase costs?
Lakebase compute and storage usage is available in system.billing.usage, with separate storage type breakdowns (BRANCH_DATA_STORAGE, BRANCH_CHANGE_STORAGE, BRANCH_HISTORY_STORAGE). Synced Table pipeline costs can also be queried from system.billing.usage by filtering on pipeline ID and joining to list_prices.
Source and freshness
- Source
- databricks.com
- Published
- Added to Yomu
Summaries are generated from the original by a language model. Check the original before relying on it.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
The best engineering reads, once a week
One useful email. Unsubscribe anytime.