Loading…
Taking AUTO CDC to the next level: Solving the hardest real-world use cases
Josh Seidel, Shanelle Roman, Sudhanva Huruli
- Source
- Databricks
- Published
- Added to Yomu
Summary
The post describes new AUTO CDC capabilities for challenges that standard CDC patterns do not fully address, including out-of-order bitemporal timelines, partial updates, and durable audit history. Bitemporal AUTO CDC tracks business and system time independently through four managed interval columns, rewriting affected history when corrections arrive out of order so either point-in-time view can be reconstructed. Partial Updates are generally available and treat selected incoming NULL values as "do not update," preventing unchanged fields from overwriting stored values. The post also describes logging business and system timestamps in MLflow for reproducibility after VACUUM, and notes that the Python API for AUTO CDC Type 1 is being contributed to Apache Spark 4.2, with a SQL interface already merged for a next release.
Context
Standard CDC patterns can struggle with out-of-order bitemporal timelines, updates that contain only changed fields, and audit or machine-learning reproducibility requirements that must survive storage retention and VACUUM.
Approach / What changed
AUTO CDC adds dual-axis business-time and system-time history tracking, automatic handling for selected NULL values in partial updates, and open-source Apache Spark support. Bitemporal tables maintain managed intervals and rewrite affected history for out-of-order events, while MLflow timestamps can identify later training datasets.
Takeaways
- Bitemporal AUTO CDC uses __START_AT and __END_AT for business time plus __SYSTEM_START_AT and __SYSTEM_END_AT for system time, allowing reconstruction along either timeline.
- AutoCDC Partial Updates is generally available and supports column lists, exceptions, or a per-row source column to determine which incoming NULL values mean "do not update."
- The contributed Apache Spark 4.2 Type 1 API includes state for early-arriving events, convergent retried microbatches, and compatibility with Delta Lake and Apache Iceberg.