Granular Usage Attribution for dbt Pipelines with Query Tags

Summary

Databricks Query Tags give dbt pipelines granular usage attribution when identical query-history rows obscure model ownership, cost, and performance. The dbt-databricks adapter version 1.11+ automatically injects model, materialization, dbt-core, and adapter-version metadata, while profile-level and model-level configurations add custom dimensions such as team, cost center, project, and environment. These tags are stored as a MAP<STRING, STRING> in system.query.history, where SQL can extract them for cost attribution, performance debugging, monitoring, and dashboards. In the reference project, four mart tables accounted for 92% of compute time, whereas staging views and metric views were near-instantaneous. The accompanying open-source project demonstrates Genie exploration, a self-monitoring AI/BI dashboard, metric-view tagging, and deployment with Databricks bundles, while recommending consistent profile hierarchies, environment tags, and limited custom metadata.

Takeaways

1. 1

   dbt-databricks 1.11+ automatically adds model name, materialization, dbt-core version, and adapter version tags to model executions.

2. 2

   Profile-level tags apply across a project, while model-level tags merge with them and override matching keys for more granular attribution.

3. 3

   In the reference project, four mart tables consumed 92% of compute time; staging views and metric views were near-instantaneous.