Open Platform, Unified Pipelines: Why dbt on Databricks is Accelerating

Summary

dbt provides structure for transforming raw data into curated datasets, but the post argues that its effectiveness depends on the underlying platform. Databricks is presented as an open lakehouse combining storage, compute, governance, orchestration, and monitoring to reduce duplicated data, fragmented permissions, and operational overhead. Lakeflow Jobs runs dbt alongside ingestion, downstream actions, retries, and dependencies, while Unity Catalog supplies persistent permissions, documentation, lineage, and fine-grained controls. The integration also supports Delta Lake and Apache Iceberg, query tags for cost tracking, Photon, Predictive Optimization, Liquid Clustering, and dbt materialized views. The post concludes these capabilities provide openness, integrated operations, governance, and price-performance for dbt workflows, citing one customer’s consolidation of notebooks and 7+ source systems and more than 2,900 customers already running dbt on Databricks.

Takeaways

1. 1

   dbt models on Databricks can use the open table formats Delta Lake and Apache Iceberg, while Unity Catalog supports governed reads and writes from external engines.

2. 2

   Lakeflow Jobs treats dbt as a first-class task and can combine Auto Loader ingestion, dbt transformations, dashboard refreshes, and ML retraining with unified retries and dependency management.

3. 3

   Unity Catalog permissions managed at the schema level persist across dbt table rebuilds, while row-level filters, column masks, and attribute-based access control apply across dbt, BI tools, and notebooks.