Topic

Apache Hudi

1 post about Apache Hudi. Every summary links to the original.

See the idea

Apache Hudi explainers

Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.

Newest first

Uber
Fast Copy-On-Write within Apache Parquet for Data Lakehouse ACID Upserts
Large-scale lakehouse upserts can become slow and resource-intensive, especially in Apache Hudi copy-on-write mode, where affected files are read, modified, and rewritten in full. The approach adds a Parquet row-level secondary index mapping records to files and row IDs, allowing updates to locate the specific data pages containing changed records. During copy-on-write, related pages are updated while unrelated pages are copied as byte buffers, avoiding decompression, re-encoding, and record reassembly for skipped data. Benchmarks using TPC-DS sales data, Spark jobs with matching vCore and memory settings, and update rates from 5% to 50% found significantly faster speeds than traditional approaches, including Delta Lake’s default configuration. The authors plan to integrate the row-level index and fast copy-on-write into Apache Hudi for use cases such as incremental ingestion, data deletion, and CDC.

The best engineering reads, once a week.

Topic

1 post about Apache Hudi. Every summary links to the original.

See the idea

Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.

Newest first

Uber
Fast Copy-On-Write within Apache Parquet for Data Lakehouse ACID Upserts
Large-scale lakehouse upserts can become slow and resource-intensive, especially in Apache Hudi copy-on-write mode, where affected files are read, modified, and rewritten in full. The approach adds a Parquet row-level secondary index mapping records to files and row IDs, allowing updates to locate the specific data pages containing changed records. During copy-on-write, related pages are updated while unrelated pages are copied as byte buffers, avoiding decompression, re-encoding, and record reassembly for skipped data. Benchmarks using TPC-DS sales data, Spark jobs with matching vCore and memory settings, and update rates from 5% to 50% found significantly faster speeds than traditional approaches, including Delta Lake’s default configuration. The authors plan to integrate the row-level index and fast copy-on-write into Apache Hudi for use cases such as incremental ingestion, data deletion, and CDC.

The best engineering reads, once a week.