Loading post…
DatabricksRyan Blue, Daniel Weeks, Jason Reid, Benjamin Mathew, Hao JiangAnnouncement
The next era of the open lakehouse: Apache Iceberg™ v3 in Public Preview on Databricks
Summary
Databricks has placed support for Apache Iceberg v3 into Public Preview, bringing new table-format capabilities to its open lakehouse. Unity Catalog Managed Iceberg v3 tables support Row Lineage, Deletion Vectors, and the VARIANT type, while interoperability extends these features to foreign Iceberg tables. Row IDs and modification sequence numbers identify changed rows, and lightweight delete files avoid immediate Parquet rewrites, with data manipulation reported as up to 10x faster than copy-on-write. VARIANT stores evolving semi-structured payloads beside relational columns for direct SQL querying, while Unity Catalog provides cross-catalog governance and Delta-Iceberg interoperability without data duplication. Iceberg v3 is available on Databricks Runtime 18.0+ with Unity Catalog enabled; Databricks is also advancing Iceberg v4 metadata proposals.
Context
Iceberg v3 support is in Public Preview on Databricks. The source positions the release as a way to combine Iceberg and Delta Lake capabilities, support incremental processing and semi-structured data, and preserve interoperability across engines and catalogs.
Approach / What changed
The implementation combines Row Lineage, Deletion Vectors, and VARIANT in Unity Catalog Managed Iceberg v3 tables. Row IDs and modification sequence numbers identify changed rows, while delete files record logical deletions without immediately rewriting Parquet data. VARIANT stores semi-structured payloads with relational columns, and Unity Catalog provides governance and interoperability across catalogs and engines.
Takeaways
Row Lineage assigns every row a permanent row ID and modification sequence number, allowing pipelines to identify changed rows without relying on full table scans or external CDC systems.
Deletion Vectors record logically deleted rows in lightweight delete files rather than immediately rewriting large Parquet files; the source reports data manipulation up to 10x faster than copy-on-write.
VARIANT stores evolving semi-structured data alongside relational columns and supports direct SQL queries, avoiding required flattening, separate storage, or normalization ETL.
FAQ
How do Row Lineage and Deletion Vectors support incremental processing?
Row Lineage identifies changed rows through permanent row IDs and modification sequence numbers. Deletion Vectors track logical deletions in separate lightweight files, so pipelines can process changes without full scans or immediate large-file rewrites.
What problem does Iceberg v3 VARIANT solve?
VARIANT stores semi-structured payloads beside relational columns in the same Iceberg table and makes them queryable with standard SQL. New fields and event types can be queried without a schema migration, flattening step, or separate normalization system.
What is required to use Iceberg v3 on Databricks?
Iceberg v3 is available in Public Preview on Databricks Runtime 18.0 or later. Unity Catalog must be enabled.
Source and freshness
- Source
- databricks.com
- Published
- Added to Yomu
Summaries are generated from the original by a language model. Check the original before relying on it.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
The best engineering reads, once a week
One useful email. Unsubscribe anytime.