How to Make Apache Hudi Copy-On-Write Upserts Less Wasteful
Learn how Apache Hudi copy-on-write upserts can use row-level file and page locations to update changed data while avoiding unnecessary decompression and rewriting.
Loading post…
UberBenchmark
Fast Copy-On-Write within Apache Parquet for Data Lakehouse ACID Upserts
Summary
Large-scale lakehouse upserts can become slow and resource-intensive, especially in Apache Hudi copy-on-write mode, where affected files are read, modified, and rewritten in full. The approach adds a Parquet row-level secondary index mapping records to files and row IDs, allowing updates to locate the specific data pages containing changed records. During copy-on-write, related pages are updated while unrelated pages are copied as byte buffers, avoiding decompression, re-encoding, and record reassembly for skipped data. Benchmarks using TPC-DS sales data, Spark jobs with matching vCore and memory settings, and update rates from 5% to 50% found significantly faster speeds than traditional approaches, including Delta Lake’s default configuration. The authors plan to integrate the row-level index and fast copy-on-write into Apache Hudi for use cases such as incremental ingestion, data deletion, and CDC.
Context
Apache Hudi, Delta Lake, and Apache Iceberg support lakehouse use cases such as incremental ingestion, data retention, and change data capture. Copy-on-write can become costly as data volume grows because files containing updated records are rewritten, and the source cites cases where updates affected 90 percent of files and involved around 100 TB of rewrites.
Approach / What changed
The implementation introduces a Parquet row-level secondary index that maps a record to its file and row ID, then uses Parquet metadata to identify the affected data pages. Pages in scope are updated, while unrelated pages are copied byte-for-byte into the new file without decompression, re-encoding, or record reassembly.
Takeaways
The row-level secondary index maps records to file and row ID, enabling updates to locate affected Parquet pages instead of only locating files.
Partial copy-on-write updates only related data pages and copies unrelated pages as byte buffers, avoiding expensive decompression, re-encoding, and record reassembly.
TPC-DS benchmarking with 5% to 50% of data updated reported significantly faster speeds than traditional approaches, including Delta Lake with default configurations.
FAQ
How does the row-level index reduce copy-on-write work?
It maps each record to a file and row ID, which identifies the Parquet data pages containing the records to update. Other pages can then be skipped during modification and copied unchanged.
What does partial copy-on-write do with unrelated pages?
It copies unrelated pages as byte buffers into the new file without decompressing, re-encoding, or reassembling their records. Only pages within the update scope are changed.
How was the performance comparison conducted?
The tests used TPC-DS sales data and Spark jobs with the same vCore and memory settings. They compared update rates from 5% to 50% against Delta Lake using its default out-of-box configuration.
Source and freshness
- Source
- uber.com
- Published
- Added to Yomu
Summaries are generated from the original by a language model. Check the original before relying on it.
See the idea
Visual explainers
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
The best engineering reads, once a week
One useful email. Unsubscribe anytime.