What is an open lakehouse? Open data standards, explained.

Summary

The piece defines an open lakehouse as a lakehouse whose storage, table format, processing engine, catalog, and ML and AI tooling use open standards and remain interchangeable. It contrasts this architecture with warehouses, lakes, and proprietary lakehouses, emphasizing low-cost object storage, ACID transactions, governance, schema guarantees, and the ability to change engines without rewriting data. Its reference stack combines open table formats such as Delta Lake and Apache Iceberg with Apache Parquet, Apache Spark, Unity Catalog, and MLflow, while allowing engines including DuckDB, Trino, and PyIceberg to work on the same data. The article also distinguishes open standards from open-source code, explains that a table format is only one layer of the stack, and states that the components can be self-hosted or consumed through a managed service.

Context

The article clarifies what open means in the lakehouse context and distinguishes open lakehouses from data lakes, data warehouses, proprietary lakehouses, open-source projects, and open table formats. It addresses concerns about vendor lock-in, portability, governance, multi-engine access, and the operational choice between self-hosting and managed services.

Approach / What changed

It presents a layered reference architecture using open table and file formats, open processing engines, unified governance, and ML and AI tooling. It explains the roles of Delta Lake, Apache Iceberg, Apache Parquet, Apache Spark, Unity Catalog, and MLflow, and describes how compatible engines can work on the same data without requiring another copy.

Takeaways

• An open lakehouse requires openness across the architecture, including storage, table format, engine, catalog, and ML and AI tooling; an open table format alone is only one layer.
• Open table formats such as Delta Lake and Apache Iceberg provide ACID transactions, schema evolution, and time travel on files in object storage, while engines such as Spark, DuckDB, Trino, and PyIceberg can work with the data.
• Self-hosting is supported because the components run on users’ infrastructure; this avoids licensing costs but requires users to provide object storage, compute, and operational effort.

Related reading