---
title: "dbt"
description: "5 posts about dbt, summarised, each linking to the original."
---

# dbt
> 5 posts about dbt, summarised, each linking to the original.

## Articles

### [Scaling for MHHS: 50x cost-efficient margin data engineering at Octopus Energy](https://yomu.fyi/post/scaling-for-mhhs-50x-cost-efficient-margin-data-engineering-at-octopus.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Saad Ali, David Poulet, Daniel Taylor, Ismail Makhlouf
- Published: May 23, 2026

Octopus Energy’s move to Market-wide Half-Hourly Settlement (MHHS) increases settlement data volume 48x for its more than 8 million customers, exposing the cost limits of a monthly, single-grain margin pipeline. The rebuild separates margin processing into half-hourly settlement cost, half-hourly revenue for smart tariffs, and daily revenue for standard tariffs, all backed by a unified multi-grain consumption layer. A “Job of Jobs” Databricks workflow coordinates the streams, while Change Data Feed where available and microbatch dbt elsewhere enable incremental processing; Spark work uses pruning, broadcast joins, partition tuning, and Adaptive Query Execution. Rows processed per run fell from 25 billion to 300 million, and freshness improved from weekly to daily. Projected cost per settlement date fell from $23.63 to $0.48, with approximately $1 million in annualised cost avoidance, excluding upstream savings.


### [Open Platform, Unified Pipelines: Why dbt on Databricks is Accelerating](https://yomu.fyi/post/open-platform-unified-pipelines-why-dbt-on-databricks-is-accelerating.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Srilekha Dornadula, Ramiz Bozai
- Published: Apr 16, 2026

dbt provides structure for transforming raw data into curated datasets, but the post argues that its effectiveness depends on the underlying platform. Databricks is presented as an open lakehouse combining storage, compute, governance, orchestration, and monitoring to reduce duplicated data, fragmented permissions, and operational overhead. Lakeflow Jobs runs dbt alongside ingestion, downstream actions, retries, and dependencies, while Unity Catalog supplies persistent permissions, documentation, lineage, and fine-grained controls. The integration also supports Delta Lake and Apache Iceberg, query tags for cost tracking, Photon, Predictive Optimization, Liquid Clustering, and dbt materialized views. The post concludes these capabilities provide openness, integrated operations, governance, and price-performance for dbt workflows, citing one customer’s consolidation of notebooks and 7+ source systems and more than 2,900 customers already running dbt on Databricks.


### [Cost Efficient Snowflake CI](https://yomu.fyi/post/cost-efficient-snowflake-ci.md)
- Company: [Ramp](https://yomu.fyi/company/ramp.md)
- Author: Kevin Chao
- Published: Nov 28, 2022

Ramp’s data team used Snowflake CI checks that created a full duplicate production database for each pull request, causing rising warehouse costs and up to 30 minutes of developer idle time as pull-request volume and model count grew. They changed the process by storing dbt manifest.json artifacts in S3, using dbt’s state selector to build modified models, and generating an upstream-model list for cloning. A custom dbt macro uses Snowflake’s information schema to create zero-copy clones of required production tables in the pull-request database, preserving a single namespace for debugging instead of relying on defer across databases. The resulting CI command sequence builds and tests modified models plus direct downstream dependencies, and the post reports a dramatic, mostly sustained dip in the rolling seven-day average warehouse cost despite higher pull-request velocity.


### [How to Build a Production Grade Workflow with SQL Modelling](https://yomu.fyi/post/how-to-build-a-production-grade-workflow-with-sql-modelling.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Nov 19, 2020

Shopify’s Starscream data platform handled 76,000 jobs and wrote 300 terabytes daily, but its PySpark workflow was slow and cumbersome for commonplace reporting. An audit found that about 70 percent of PySpark jobs were full batch queries that did not require generalized computing, while users often wrote SQL before translating it into Python. The team built Seamster with dbt and Google BigQuery, adding source and model structures, ownership metadata, Python-based unit tests with fixed mock inputs, CI validation, and warehouse rules. A base layer isolates raw-source changes, while models are organized into base, application-ready, and presentation layers and defined through SQL, schema, README, and test files. After a multiweek beta, most users reported shipping models in days rather than weeks, although dbt’s incremental support could not safely handle some late-arriving-data, key-resolution, rebuild, and very large-model cases.


### [How to Track State with Type 2 Dimensional Models](https://yomu.fyi/post/how-to-track-state-with-type-2-dimensional-models.md)
- Company: [Shopify](https://yomu.fyi/company/shopify.md)
- Author: 2023-10-18
- Published: Aug 27, 2020

Application databases often retain only current values in Type 1 dimensions, limiting analysis of historical settings such as feature adoption, retention, and switching behavior. The post explains Type 2 dimensional models through Shopify’s need to track users’ admin languages over time, contrasting application-model changes, scheduled database snapshots, and event logging. Its selected implementation uses Rails after\_commit callbacks to send created or updated records to Kafka, then transforms that event history into records with valid\_from, valid\_to, and is\_current fields using ETL recipes involving PySpark and dbt. The approach provides the required granularity but can miss changes or events, requires delete handling and data-quality checks, and is presented as an iterative solution; MySQL binlogs are described as a more reliable future source.