---
title: "Stop hand-coding change data capture pipelines"
description: "Hand-coded change data capture (CDC) and slowly changing dimension (SCD) pipelines become fragile as they handle out-of-order updates, duplicate events, deletes, late-arriving data, retries, backfills, and schema evolution. The post presents AutoCDC in Lakeflow Spark Declarative Pipelines as a declarative alternative: engineers specify keys, sequencing, delete conditions, and SCD Type 1 or Type 2 semantics while the platform manages ordering, state, deduplication, versioning, and incremental processing. Examples show a current-state table that keeps Mercedes’s latest city and removes deleted user 123, alongside a Type 2 table that preserves validity windows through __START_AT and __END_AT. The supplied comparison reports roughly 6–10 lines of declarative definition versus 40–200+ lines of custom logic, and says AutoCDC supports both change data feeds and snapshot-based CDC."
---

# Stop hand-coding change data capture pipelines

[Databricks](https://yomu.fyi/company/databricks) · Matt Jones, Zoé Durand, Phoebe Weiser, Bilal Aslam, Ray Zhu · Apr 22, 2026

**Type:** Problem & solution

## Summary

Hand-coded change data capture (CDC) and slowly changing dimension (SCD) pipelines become fragile as they handle out-of-order updates, duplicate events, deletes, late-arriving data, retries, backfills, and schema evolution. The post presents AutoCDC in Lakeflow Spark Declarative Pipelines as a declarative alternative: engineers specify keys, sequencing, delete conditions, and SCD Type 1 or Type 2 semantics while the platform manages ordering, state, deduplication, versioning, and incremental processing. Examples show a current-state table that keeps Mercedes’s latest city and removes deleted user 123, alongside a Type 2 table that preserves validity windows through \_\_START\_AT and \_\_END\_AT. The supplied comparison reports roughly 6–10 lines of declarative definition versus 40–200+ lines of custom logic, and says AutoCDC supports both change data feeds and snapshot-based CDC.

## Context

CDC and SCD pipelines are difficult to implement and maintain by hand because they must correctly handle updates, deletes, duplicate and out-of-order events, late-arriving data, retries, reprocessing, backfills, and schema evolution. Snapshot sources can also require separate logic when native change data feeds are unavailable.

## Approach / What changed

AutoCDC in Lakeflow Spark Declarative Pipelines provides a declarative abstraction for CDC and SCD patterns. Pipeline definitions declare keys, sequencing columns, delete conditions, and whether data is stored as SCD Type 1 or Type 2; the platform handles ordering, deduplication, history management, and incremental processing for change-feed and snapshot-based sources.

## Takeaways

- AutoCDC supports SCD Type 1 current-state tables and SCD Type 2 historical tables, including delete handling and version management.
- For change data feeds, a declared sequencing column lets AutoCDC apply out-of-sequence records correctly while handling deduplication, late-arriving data, and incremental processing.
- The post compares approximately 6–10 lines of declarative AutoCDC definition with 40–200+ lines of hand-written CDC or snapshot pipeline logic.

**Tags:** [Data Pipelines](https://yomu.fyi/topic/data-pipelines), [Delta Lake](https://yomu.fyi/topic/delta-lake), [Spark](https://yomu.fyi/topic/spark)

- Source: [Databricks](https://www.databricks.com/blog/stop-hand-coding-change-data-capture-pipelines)
- Source URL: https://www.databricks.com/blog/stop-hand-coding-change-data-capture-pipelines
- Ingested by Yomu: 2026-08-31T03:41:44.504Z

[Read original post](https://www.databricks.com/blog/stop-hand-coding-change-data-capture-pipelines)