How to get the most out of running dbt on Databricks

Summary

Tens of thousands of organizations use dbt, and where it runs directly affects pipeline cost, speed, and operational complexity. Running dbt on Databricks provides an open, unified platform, yet many teams underutilize its capabilities. The post details practices to reduce costs, such as applying appropriate incremental strategies and replacing full recomputes with materialized views, which delivered a 17x speedup in a benchmark. It also covers automating data layout with Liquid Clustering and Predictive Optimization, and attributing every dollar to specific models via query tags. For operational simplicity, it recommends bringing ingestion into dbt with streaming tables, standardizing metrics through Unity Catalog Semantics, and orchestrating pipelines natively with Lakeflow Jobs.

Takeaways

1. 1

   Choosing the right incremental strategy per model reduces full table rebuilds; replace_where is more efficient than merge when changes can be described with a predicate.

2. 2

   Materialized views on Databricks automate incrementalization with a one-line config change and delivered a 17x speedup in a TPC-DI benchmark at the serving layer.

3. 3

   Lakeflow Jobs collapse ingestion, dbt, and downstream tasks into a single native DAG with built-in scheduling, retries, and alerts, removing the need for a third-party orchestrator.

FAQ