Debunking 8 data layout myths: why Liquid Clustering outperforms partitioning

Summary

Partitioning has long been the standard physical layout for Hadoop- and Hive-era processing, but the post argues that changing Lakehouse workloads make fixed choices brittle. Liquid Clustering treats keys as guidance for file organization, allowing layouts to evolve, supporting low- and high-cardinality filters, maintaining file sizes, and enabling row-level concurrency. Using transaction-log metadata and per-column statistics, the post says modern Delta and Iceberg engines prune files rather than directories, and reports 35% lower clustering time, 22% faster queries, roughly 90% faster metadata-only deletes, and 23-minute planning for a 10-PB table. Conversion examples include Bolt’s 138% higher write throughput and up to 63% faster reads, while a 1.1-PB workload saw 5.9x faster queries, 86% fewer bytes read, and a 27% smaller table after clustering by date, hour, source, and id.

Context

Hive-style partitioning requires choosing a physical organization when a table is created. The post says changing Lakehouse workloads, including agent-generated data and shifting query patterns, make incorrect partition choices costly, leading to over-partitioning, small files, data skew, and table rewrites.

Approach / What changed

Liquid Clustering uses clustering keys to guide file organization rather than fixing the layout in directory partitions. The post describes changing keys, Automatic Liquid Clustering, file-level pruning through transaction-log statistics, low-cardinality optimizations, row-level concurrency, and conversion of partitioned tables to Liquid in place.

Takeaways

• Delta and Iceberg query planning uses transaction logs and per-file column statistics for pruning; the post says directory structure does not provide a separate directory-level shortcut.
• Liquid Clustering benchmarks reported 35% lower clustering time and 22% faster queries for a low-cardinality optimization, plus roughly 90% faster metadata-only DELETE operations and up to 27x faster metadata-only aggregates.
• On a 1.1 PB workload, clustering by date, hour, source, and id reduced wall-clock time from 406 seconds to 70 seconds, cut bytes read by 86%, and reduced total table size by 27%.

Related reading