The Convergence of Open Table Formats and Open Catalogs: Catalog Commits is Generally Available

Summary

Catalog Commits is generally available for Unity Catalog managed Delta tables, aligning Delta with Iceberg’s catalog-oriented model and making the catalog responsible for table discovery, access, and latest table state. The change addresses three coordination problems: metadata “split brain” when engines write directly to storage, fragmented multi-engine governance, and the historical inability to coordinate atomic writes across multiple tables. With Catalog Commits enabled, Unity Catalog brokers table access through standardized APIs, keeping catalog and table state synchronized and enabling consistent authorization, holistic auditability, automated optimizations, and multi-statement, multi-table ACID transactions on Databricks. The release supports Databricks products and engines including Delta Spark, Delta Flink, Starburst Trino, DuckDB, and StreamNative, while Delta Kernel provides a shared path for connector support.

Takeaways

1. 1

   Catalog Commits keeps catalog metadata and Delta table state synchronized by routing engine access through common APIs, addressing silent metadata drift caused by direct object-storage writes.

2. 2

   Unity Catalog’s centralized access layer standardizes table discovery, authorization, auditing, and row- or column-level governance across engines and agents instead of relying on hardcoded storage paths.

3. 3

   Databricks and Unity Catalog can coordinate atomic writes across multiple tables, adding multi-table ACID semantics and enabling traditional warehousing workloads on the lakehouse.