Databricks at SIGMOD 2026

Summary

Databricks says its work on Spark Declarative Pipelines will be featured at SIGMOD 2026, where it received an honorable mention award, and that it will attend June 1–5 in Bangalore as a Platinum Sponsor. The announcement describes two mix-and-match incremental programming paths in SDP: materialized views maintained by the Enzyme engine and streaming APIs with stateful operators and watermarks. Enzyme applies incremental view maintenance to ETL, supporting production materialized views with joins, window functions, aggregations, combinations of these patterns, non-deterministic functions such as current_date(), and AI-specific functions. It also supports Python as well as SQL, detects changes in materialized-view definitions, chooses partition- or row-level updates, selectively caches intermediate results, and uses a cost model based on plan information and prior executions; the post says it outperforms an anonymized competing solution.

Takeaways

1. 1

   Enzyme extends incremental materialized-view maintenance to ETL workloads, including joins, window functions, aggregations, non-deterministic functions, and AI-specific functions.

2. 2

   Spark Declarative Pipelines supports two incremental programming styles: materialized views maintained by Enzyme and streaming APIs with stateful operators and watermarks.

3. 3

   Enzyme uses language-aware change detection, selective intermediate-result caching, partition-level update decisions, and a cost model based on plan information and prior executions.