Apache Iceberg replaces Hive's partition-based metadata with a versioned catalog layer, trading operational simplicity for ACID guarantees, schema evolution, and cross-engine compatibility.
Apache Iceberg and Hive represent two approaches to organizing petabyte-scale data lakes. Hive relies on partition directories and a shared metastore, requiring manual partition registration and vulnerable to stale metadata when engines write directly to storage. Iceberg introduces a hidden metadata tree—snapshots, manifests, and data files—that the catalog controls, enabling atomic multi-table transactions, time travel, and fine-grained governance without format-specific prefixes. The trade-off: Iceberg adds metadata overhead and catalog dependence, but eliminates small-file fragmentation, catalog listing latency, and the split-brain problem where storage and metadata diverge. Teams migrating petabyte lakes often prioritize high-value tables first, using tools like Z-ordering to cut query time by 90% while reducing S3 costs.
The posts behind it
Databricks · · ExplainerRippling describes an architecture in which Databricks writes some tables while Trino, Snowflake, and other lakehouse engines read them, making catalog ownership a design decision rather than a metadata detail. Its rule is that the catalog follows the writer: AWS-native workloads use Glue, Databricks workloads use Unity Catalog, and Snowflake-produced shared data uses Horizon Catalog with Snowflake-managed Iceberg tables. For selected Databricks outputs, Unity Catalog-managed Apache Iceberg stores table data and metadata in Rippling-owned S3, exposes access through the Iceberg REST Catalog API or federation, and uses credential vending for scoped storage access. This lets downstream engines read Parquet directly without duplicate materialization or producer-side query execution. Databricks also provides Predictive Optimization for maintenance, while the trade-off is continued Unity Catalog dependence for metadata, permissions, lineage, and lifecycle.
Grab · · Problem & solutionGrab is transitioning its petabyte-scale data lake from Hive Parquet tables managed through the Hive Metastore to a table-centric architecture centered on Apache Iceberg, while retaining storage-compute separation on S3. The move addresses catalog listing latency, severe small-file fragmentation, manual partition registration, absent native ACID operations, and catalog state becoming stale after direct S3 changes. Rather than converting the entire lake at once, Grab prioritizes high-value tables and uses features such as Z-ordering; representative workloads cut one query from 70 seconds to 6, reduced daily S3 API costs by up to 95%, and halved cluster resource usage. Its open-sourced UnifiedSparkCatalog detects table formats through metastore metadata or path inference and routes operations to the appropriate Spark catalog without format-specific prefixes. Deployment also required resolving Hive lock contention, timestamp incompatibilities, and storage-tier cost spikes.
Databricks · · ExplainerThe piece defines an open lakehouse as a lakehouse whose storage, table format, processing engine, catalog, and ML and AI tooling use open standards and remain interchangeable. It contrasts this architecture with warehouses, lakes, and proprietary lakehouses, emphasizing low-cost object storage, ACID transactions, governance, schema guarantees, and the ability to change engines without rewriting data. Its reference stack combines open table formats such as Delta Lake and Apache Iceberg with Apache Parquet, Apache Spark, Unity Catalog, and MLflow, while allowing engines including DuckDB, Trino, and PyIceberg to work on the same data. The article also distinguishes open standards from open-source code, explains that a table format is only one layer of the stack, and states that the components can be self-hosted or consumed through a managed service.
Databricks · · AnnouncementDatabricks announces a broad set of Apache Iceberg capabilities in Unity Catalog, spanning General Availability, previews, and beta releases. Managed Iceberg is GA, supporting table creation, reads, writes, optimization, governance, and sharing, while Iceberg v3 adds deletion vectors, row tracking, and VARIANT across managed, foreign, and UniForm-enabled tables. Unity Catalog also federates external catalogs, vends credentials, shares live data with Iceberg REST-compatible clients through Delta Sharing, and applies attribute-based access control during server-side scan planning for supported external engines. These capabilities are presented as a unified approach to open APIs, cross-engine governance, zero-copy sharing, and production performance without copying data. The post also outlines Iceberg v4 and a proposal for Delta 5.0 to adopt an adaptive metadata tree structure.
Databricks · · AnnouncementCatalog Commits is generally available for Unity Catalog managed Delta tables, aligning Delta with Iceberg’s catalog-oriented model and making the catalog responsible for table discovery, access, and latest table state. The change addresses three coordination problems: metadata “split brain” when engines write directly to storage, fragmented multi-engine governance, and the historical inability to coordinate atomic writes across multiple tables. With Catalog Commits enabled, Unity Catalog brokers table access through standardized APIs, keeping catalog and table state synchronized and enabling consistent authorization, holistic auditability, automated optimizations, and multi-statement, multi-table ACID transactions on Databricks. The release supports Databricks products and engines including Delta Spark, Delta Flink, Starburst Trino, DuckDB, and StreamNative, while Delta Kernel provides a shared path for connector support.
More like this
AWS Glue runs serverless Spark jobs that transform data in batches; engine-native pipelines transform during ingestion. This comparison shows how to pick between them by freshness, cost, and data fidelity.
Compare two mental models for brand: a static artifact (logos, guidelines, centralized control) and a living operating system (values, APIs, support, trust). See which choices fit your engineering team.
A side-by-side comparison of the two accountability models engineers actually rely on: a named human owner per outcome, and system-enforced traces, budgets, and isolation.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
One useful email. Unsubscribe anytime.
