Loading post…
DatabricksFernando Muñoz, Ludwig Brummer, Maxim HammerProblem & solution
Scaling and Operating a Large dbt Project on Databricks: IFCO's Data Team on Performance, Visibility, and Debugging
Summary
IFCO’s data platform team, with Databricks Forward Deployed Engineering, optimized a large dbt project on Databricks to handle billions of daily tracking events for reusable packaging assets. The pipeline faces scale, unpredictable late-arriving data, and the need to reconcile history when late events change previously reported asset timelines. The team tuned incremental model settings to match Delta Lake write behaviors, including liquid clustering, deliberate merge vs. delete+insert strategies, and bounding writes to recent ingestion windows. Diagnosing performance from actual query plans rather than compiled SQL revealed that the consolidation model was scanning far more rows than necessary. These changes cut the core semantic layer job’s daily runtime by over 60 percent, retired a costly nightly full refresh, and reduced daily compute cost by 58 percent.
Context
IFCO operates a large reusable packaging pool, tracking billions of raw events per day for KPIs like cycle time and loss. The data pipeline faces scale, unpredictable late-arriving data, and the need for historic reconciliation when late events change previously reported asset timelines.
Approach / What changed
IFCO’s data team, with Databricks Forward Deployed Engineering, optimized their dbt-on-Databricks pipeline by tuning incremental model settings to match Delta Lake write behaviors: liquid clustering on asset and event date, deliberate choice of merge vs. delete+insert strategies, bounding writes to a recent ingestion window, and scoping recomputation to only changed assets. They diagnosed performance from actual query plans rather than compiled SQL, and deployed per-model task graphs for visibility and safe reruns.
Takeaways
Tuning dbt incremental model settings to match Delta Lake write behaviors—liquid clustering, merge vs. delete+insert, and time-bounded writes—can dramatically reduce rows scanned and runtime.
Diagnosing performance from the actual executed query plan, not the compiled SQL, is essential because dbt generates a larger operation behind the scenes that reveals real bottlenecks.
Scoping recomputation to only assets touched by new or late data, identified from an ingestion watermark, cut the assets recomputed per run from almost the full pool to about 3-5%.
FAQ
How does bounding the write to a recent window work?
A time-bound predicate restricts the write to a recent window based on when a row was ingested, not when the event happened. This ensures the merge matches against a recent slice of the destination table rather than the whole table. The window must be wide enough to cover the gap between data landing and job processing; set too narrow, late data is silently skipped.
Why is the actual query plan more useful than compiled SQL for debugging?
The only way to find where time and memory go is to read the actual executed query plan, stage by stage, from query history, not the compiled SQL. dbt compile renders the model's SELECT, but for an incremental model, dbt generates a larger operation including temporary views, scans of the destination table, and the final write. The actual plan reveals the real bottlenecks.
Source and freshness
- Source
- databricks.com
- Published
- Added to Yomu
Summaries are generated from the original by a language model. Check the original before relying on it.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
The best engineering reads, once a week
One useful email. Unsubscribe anytime.