Advancing Apache Iceberg on Databricks: Iceberg v3 GA, Open Sharing, and Unified Governance

Summary

Databricks announces a broad set of Apache Iceberg capabilities in Unity Catalog, spanning General Availability, previews, and beta releases. Managed Iceberg is GA, supporting table creation, reads, writes, optimization, governance, and sharing, while Iceberg v3 adds deletion vectors, row tracking, and VARIANT across managed, foreign, and UniForm-enabled tables. Unity Catalog also federates external catalogs, vends credentials, shares live data with Iceberg REST-compatible clients through Delta Sharing, and applies attribute-based access control during server-side scan planning for supported external engines. These capabilities are presented as a unified approach to open APIs, cross-engine governance, zero-copy sharing, and production performance without copying data. The post also outlines Iceberg v4 and a proposal for Delta 5.0 to adopt an adaptive metadata tree structure.

Context

The open lakehouse needs catalog capabilities that preserve interoperability across engines while providing consistent governance, performance, and data sharing. The post identifies open APIs, external catalog federation, cross-engine governance, secure sharing, and continuous performance and format innovation as operational requirements.

Approach / What changed

Databricks is expanding Unity Catalog with managed and foreign Iceberg support, Iceberg REST Catalog APIs, credential vending, catalog federation, cross-engine ABAC, Delta Sharing for Iceberg clients, materialized views, and automatic optimization. It is also advancing Iceberg v3 and proposing an adaptive metadata tree structure for Iceberg v4 and Delta 5.0.

Takeaways

• Managed Iceberg is generally available in Unity Catalog, with Predictive Optimization and Liquid Clustering for table performance.
• Cross-engine ABAC evaluates column masks, row filters, and tag-based policies during server-side scan planning through Iceberg REST Catalog Scan APIs.
• Iceberg v3 support includes deletion vectors, row tracking, and VARIANT across managed, foreign, and UniForm-enabled tables.

Related reading