Granular Usage Attribution for dbt Pipelines with Query Tags

Summary

Databricks Query Tags give dbt pipelines granular usage attribution when identical query-history rows obscure model ownership, cost, and performance. The dbt-databricks adapter version 1.11+ automatically injects model, materialization, dbt-core, and adapter-version metadata, while profile-level and model-level configurations add custom dimensions such as team, cost center, project, and environment. These tags are stored as a MAP<STRING, STRING> in system.query.history, where SQL can extract them for cost attribution, performance debugging, monitoring, and dashboards. In the reference project, four mart tables accounted for 92% of compute time, whereas staging views and metric views were near-instantaneous. The accompanying open-source project demonstrates Genie exploration, a self-monitoring AI/BI dashboard, metric-view tagging, and deployment with Databricks bundles, while recommending consistent profile hierarchies, environment tags, and limited custom metadata.

Context

The dbt project runs 80 nightly models, but query history labels them identically as “Databricks Dbt,” making it difficult to identify model ownership, attribute warehouse costs, compare performance, and assess optimization effects.

Approach / What changed

Configure Query Tags through dbt-databricks at profile and model levels, use automatically injected dbt metadata alongside custom business dimensions, then query system.query.history to attribute compute, debug performance, and build monitoring dashboards. The reference project also covers metric views, Genie, and Databricks bundle deployment.

Takeaways

• dbt-databricks 1.11+ automatically adds model name, materialization, dbt-core version, and adapter version tags to model executions.
• Profile-level tags apply across a project, while model-level tags merge with them and override matching keys for more granular attribution.
• In the reference project, four mart tables consumed 92% of compute time; staging views and metric views were near-instantaneous.

Related reading