GrabRahul PentiProblem & solution
Grab is transitioning its petabyte-scale data lake from Hive Parquet tables managed through the Hive Metastore to a table-centric architecture centered on Apache Iceberg, while retaining storage-compute separation on S3. The move addresses catalog listing latency, severe small-file fragmentation, manual partition registration, absent native ACID operations, and catalog state becoming stale after direct S3 changes. Rather than converting the entire lake at once, Grab prioritizes high-value tables and uses features such as Z-ordering; representative workloads cut one query from 70 seconds to 6, reduced daily S3 API costs by up to 95%, and halved cluster resource usage. Its open-sourced UnifiedSparkCatalog detects table formats through metastore metadata or path inference and routes operations to the appropriate Spark catalog without format-specific prefixes. Deployment also required resolving Hive lock contention, timestamp incompatibilities, and storage-tier cost spikes.
Grab’s directory-based Hive Parquet architecture became a bottleneck as the data lake grew to petabytes and billions of S3 objects. The source identifies catalog latency and centralized Hive Metastore contention, small files, manual partition-management work, missing native ACID operations, and stale catalog state after direct S3 changes as the primary problems.
Grab selected Apache Iceberg as its default table format and is migrating high-value tables incrementally rather than converting the lake in one step. It also developed UnifiedSparkCatalog, which detects table formats, routes operations to the appropriate Spark catalog, supports mixed formats transparently, and falls back to the base session catalog for standard Hive tables.
UnifiedSparkCatalog initializes format-specific catalogs lazily, skips formats whose JARs are unavailable, reuses existing catalog instances, and retains the special name spark_catalog for legacy Hive DML compatibility.
Grab selected Iceberg based on community governance, engine compatibility, and long-term flexibility, while positioning the data lake to remain format-agnostic.
Migration planning accounts for one-time S3 storage-tier cost spikes by prioritizing tables according to scan frequency and API operation costs instead of migrating the entire lake at once.
Summaries are generated from the original by a language model. Check the original before relying on it.
See the idea
Apache Iceberg replaces Hive's partition-based metadata with a versioned catalog layer, trading operational simplicity for ACID guarantees, schema evolution, and cross-engine compatibility.
Keep reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
One useful email. Unsubscribe anytime.
