Loading…
Topic
dbt
8 posts about dbt. Every summary links to the original.
- 8 posts
- 4 companies
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
How to Repoint dbt ETL Pipelines to Databricks
Repointing a dbt project to Databricks involves swapping the adapter and profile, handling SQL dialect differences, and keeping transformation logic unchanged. The dbt-databricks adapter bridges the project to Databricks Lakehouse, which uses a three-level namespace (catalog.schema.table) and supports open formats like Delta Lake and Apache Iceberg. Common dialect gaps include REGEXP_SUBSTR, DIV0, and OBJECT_CONSTRUCT, which can be fixed with Databricks-native functions or dbt_utils macros; the post walks through each error and its fix. After compilation and validation using audit-helper to compare outputs model by model, the project can be deployed as a dbt task in Lakeflow Jobs with built-in governance via Unity Catalog. The process is incremental: one-time setup costs apply, then each additional model requires only dialect fixes, making it a low-risk starting point for warehouse migration.
Scaling and Operating a Large dbt Project on Databricks: IFCO's Data Team on Performance, Visibility, and Debugging
IFCO’s data platform team, with Databricks Forward Deployed Engineering, optimized a large dbt project on Databricks to handle billions of daily tracking events for reusable packaging assets. The pipeline faces scale, unpredictable late-arriving data, and the need to reconcile history when late events change previously reported asset timelines. The team tuned incremental model settings to match Delta Lake write behaviors, including liquid clustering, deliberate merge vs. delete+insert strategies, and bounding writes to recent ingestion windows. Diagnosing performance from actual query plans rather than compiled SQL revealed that the consolidation model was scanning far more rows than necessary. These changes cut the core semantic layer job’s daily runtime by over 60 percent, retired a costly nightly full refresh, and reduced daily compute cost by 58 percent.
ClickHouse is now available on the dbt platform
The ClickHouse adapter for dbt v2, dbt’s Rust rewrite, entered public beta on September 16, 2026, while ClickHouse became a natively supported data warehouse on the dbt platform in private beta. The v2 adapter ships inside the open-source dbt binary, connects through an ADBC driver using Apache Arrow, and supports both open-source ClickHouse and ClickHouse Cloud. Existing models, tests, and profiles carry over, although some features remain incomplete and dbt plans to support the v1 and v2 adapters in parallel. The platform integration provides ClickHouse connections for Studio, environments, scheduled jobs, and Catalog, while the planned path to general availability includes parity items such as ON CLUSTER DDL and distributed materializations, followed by Fusion capabilities.
Scaling for MHHS: 50x cost-efficient margin data engineering at Octopus Energy
Octopus Energy’s move to Market-wide Half-Hourly Settlement (MHHS) increases settlement data volume 48x for its more than 8 million customers, exposing the cost limits of a monthly, single-grain margin pipeline. The rebuild separates margin processing into half-hourly settlement cost, half-hourly revenue for smart tariffs, and daily revenue for standard tariffs, all backed by a unified multi-grain consumption layer. A “Job of Jobs” Databricks workflow coordinates the streams, while Change Data Feed where available and microbatch dbt elsewhere enable incremental processing; Spark work uses pruning, broadcast joins, partition tuning, and Adaptive Query Execution. Rows processed per run fell from 25 billion to 300 million, and freshness improved from weekly to daily. Projected cost per settlement date fell from $23.63 to $0.48, with approximately $1 million in annualised cost avoidance, excluding upstream savings.
Open Platform, Unified Pipelines: Why dbt on Databricks is Accelerating
dbt provides structure for transforming raw data into curated datasets, but the post argues that its effectiveness depends on the underlying platform. Databricks is presented as an open lakehouse combining storage, compute, governance, orchestration, and monitoring to reduce duplicated data, fragmented permissions, and operational overhead. Lakeflow Jobs runs dbt alongside ingestion, downstream actions, retries, and dependencies, while Unity Catalog supplies persistent permissions, documentation, lineage, and fine-grained controls. The integration also supports Delta Lake and Apache Iceberg, query tags for cost tracking, Photon, Predictive Optimization, Liquid Clustering, and dbt materialized views. The post concludes these capabilities provide openness, integrated operations, governance, and price-performance for dbt workflows, citing one customer’s consolidation of notebooks and 7+ source systems and more than 2,900 customers already running dbt on Databricks.
Cost Efficient Snowflake CI
Ramp’s data team used Snowflake CI checks that created a full duplicate production database for each pull request, causing rising warehouse costs and up to 30 minutes of developer idle time as pull-request volume and model count grew. They changed the process by storing dbt manifest.json artifacts in S3, using dbt’s state selector to build modified models, and generating an upstream-model list for cloning. A custom dbt macro uses Snowflake’s information schema to create zero-copy clones of required production tables in the pull-request database, preserving a single namespace for debugging instead of relying on defer across databases. The resulting CI command sequence builds and tests modified models plus direct downstream dependencies, and the post reports a dramatic, mostly sustained dip in the rolling seven-day average warehouse cost despite higher pull-request velocity.
How to Build a Production Grade Workflow with SQL Modelling
Shopify’s Starscream data platform handled 76,000 jobs and wrote 300 terabytes daily, but its PySpark workflow was slow and cumbersome for commonplace reporting. An audit found that about 70 percent of PySpark jobs were full batch queries that did not require generalized computing, while users often wrote SQL before translating it into Python. The team built Seamster with dbt and Google BigQuery, adding source and model structures, ownership metadata, Python-based unit tests with fixed mock inputs, CI validation, and warehouse rules. A base layer isolates raw-source changes, while models are organized into base, application-ready, and presentation layers and defined through SQL, schema, README, and test files. After a multiweek beta, most users reported shipping models in days rather than weeks, although dbt’s incremental support could not safely handle some late-arriving-data, key-resolution, rebuild, and very large-model cases.
How to Track State with Type 2 Dimensional Models
Application databases often retain only current values in Type 1 dimensions, limiting analysis of historical settings such as feature adoption, retention, and switching behavior. The post explains Type 2 dimensional models through Shopify’s need to track users’ admin languages over time, contrasting application-model changes, scheduled database snapshots, and event logging. Its selected implementation uses Rails after_commit callbacks to send created or updated records to Kafka, then transforms that event history into records with valid_from, valid_to, and is_current fields using ETL recipes involving PySpark and dbt. The approach provides the required granularity but can miss changes or events, requires delete handling and data-quality checks, and is presented as an iterative solution; MySQL binlogs are described as a more reliable future source.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.