Open Platform, Unified Pipelines: Why dbt on Databricks is Accelerating

Summary

dbt provides structure for transforming raw data into curated datasets, but the post argues that its effectiveness depends on the underlying platform. Databricks is presented as an open lakehouse combining storage, compute, governance, orchestration, and monitoring to reduce duplicated data, fragmented permissions, and operational overhead. Lakeflow Jobs runs dbt alongside ingestion, downstream actions, retries, and dependencies, while Unity Catalog supplies persistent permissions, documentation, lineage, and fine-grained controls. The integration also supports Delta Lake and Apache Iceberg, query tags for cost tracking, Photon, Predictive Optimization, Liquid Clustering, and dbt materialized views. The post concludes these capabilities provide openness, integrated operations, governance, and price-performance for dbt workflows, citing one customer’s consolidation of notebooks and 7+ source systems and more than 2,900 customers already running dbt on Databricks.

Context

The post describes data stacks that require separate systems for storage, compute, governance, orchestration, and monitoring, creating duplicated data, inconsistent permissions, fragmented observability, and ongoing performance-tuning work. It frames vendor lock-in and governance bottlenecks as additional concerns for teams scaling dbt workflows.

Approach / What changed

Databricks integrates dbt with its open lakehouse, Lakeflow Jobs, Unity Catalog, and performance features. The approach combines open table formats and interfaces with unified orchestration, persistent and fine-grained governance, lineage, cost monitoring, and automated performance optimization.

Takeaways

• dbt models on Databricks can use the open table formats Delta Lake and Apache Iceberg, while Unity Catalog supports governed reads and writes from external engines.
• Lakeflow Jobs treats dbt as a first-class task and can combine Auto Loader ingestion, dbt transformations, dashboard refreshes, and ML retraining with unified retries and dependency management.
• Unity Catalog permissions managed at the schema level persist across dbt table rebuilds, while row-level filters, column masks, and attribute-based access control apply across dbt, BI tools, and notebooks.

Related reading