One Write Path, Multiple Entries: How Rippling uses Unity Catalog-managed Apache Iceberg™

Summary

Rippling describes an architecture in which Databricks writes some tables while Trino, Snowflake, and other lakehouse engines read them, making catalog ownership a design decision rather than a metadata detail. Its rule is that the catalog follows the writer: AWS-native workloads use Glue, Databricks workloads use Unity Catalog, and Snowflake-produced shared data uses Horizon Catalog with Snowflake-managed Iceberg tables. For selected Databricks outputs, Unity Catalog-managed Apache Iceberg stores table data and metadata in Rippling-owned S3, exposes access through the Iceberg REST Catalog API or federation, and uses credential vending for scoped storage access. This lets downstream engines read Parquet directly without duplicate materialization or producer-side query execution. Databricks also provides Predictive Optimization for maintenance, while the trade-off is continued Unity Catalog dependence for metadata, permissions, lineage, and lifecycle.

Takeaways

1. 1

   Unity Catalog-managed Iceberg tables let downstream engines such as Trino, Snowflake, Spark, Athena, and EMR read the same Parquet data directly from S3 through standard Iceberg access patterns.

2. 2

   Predictive Optimization provides automatic maintenance for managed tables, including file optimization and compaction, statistics collection, and data layout optimization.

3. 3

   The data remains portable as Iceberg over Parquet in Rippling-owned S3, but moving away from Unity Catalog would require replacing its metadata, permissions, lineage, and maintenance functions.