Databricks at SIGMOD 2026

Summary

Databricks says its work on Spark Declarative Pipelines will be featured at SIGMOD 2026, where it received an honorable mention award, and that it will attend June 1–5 in Bangalore as a Platinum Sponsor. The announcement describes two mix-and-match incremental programming paths in SDP: materialized views maintained by the Enzyme engine and streaming APIs with stateful operators and watermarks. Enzyme applies incremental view maintenance to ETL, supporting production materialized views with joins, window functions, aggregations, combinations of these patterns, non-deterministic functions such as current_date(), and AI-specific functions. It also supports Python as well as SQL, detects changes in materialized-view definitions, chooses partition- or row-level updates, selectively caches intermediate results, and uses a cost model based on plan information and prior executions; the post says it outperforms an anonymized competing solution.

Context

Databricks presents Spark Declarative Pipelines as a way to simplify incremental processing for data engineering and ETL workloads. The announcement also covers the company’s SIGMOD 2026 participation and related work on streaming and incremental view maintenance.

Approach / What changed

Spark Declarative Pipelines lets customers combine materialized views, incrementally maintained by Enzyme, with streaming APIs. Enzyme supports complex materialized-view patterns, Python and SQL definitions, partition- and row-level update strategies, selective caching, and cost-based incrementalization.

Takeaways

• Enzyme extends incremental materialized-view maintenance to ETL workloads, including joins, window functions, aggregations, non-deterministic functions, and AI-specific functions.
• Spark Declarative Pipelines supports two incremental programming styles: materialized views maintained by Enzyme and streaming APIs with stateful operators and watermarks.
• Enzyme uses language-aware change detection, selective intermediate-result caching, partition-level update decisions, and a cost model based on plan information and prior executions.

Related reading