How Apache DataFusion Runs SQL In-Process
Apache DataFusion is an embeddable query engine: SQL goes in, runs as columnar Arrow batches inside your own process, and results come out without a separate service.
Loading…
Topic
Parquet
5 posts about Parquet. Every summary links to the original.
- 5 posts
- 2 companies
- 2 explainers
See the idea
Parquet explainers
How to Make Apache Hudi Copy-On-Write Upserts Less Wasteful
Learn how Apache Hudi copy-on-write upserts can use row-level file and page locations to update changed data while avoiding unnecessary decompression and rewriting.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
Loading Parquet data into MySQL with ClickHouse
The post demonstrates how to move Parquet data into MySQL and query the resulting table using ClickHouse table functions. It sets up MySQL 8.4 in Docker, runs clickhouse-local with a configuration file and named collection, and inspects StackOverflow votes stored in a 2024 Parquet file on S3. An INSERT INTO FUNCTION mysql statement loads the file into a MySQL InnoDB table with 12,296,977 rows, while subsequent examples query MySQL from ClickHouse using either translated ClickHouse syntax or an opaque MySQL SQL string. The post also shows named-collection credentials and a query that identifies posts with both upvotes and downvotes, returning ten ranked results. It concludes that this workflow stays within ClickHouse and notes that table functions also support Postgres, SQLite, and MongoDB.
ClickHouse release 26.8
ClickHouse 26.8 is a long-term support release with 98 new features, 128 performance optimizations, and 556 bug fixes. It adds background queries, PostgreSQL-style regular-expression operators, array subscripting with multiple indices, expanded Parquet and data-lake integrations, and improvements for GROUP BY and joins. Background queries return immediately while continuing after client disconnection, with progress exposed through system.processes and completion details recorded in system.query_log; the example exports 243,619,704 rows to Parquet. Performance work includes dictionary-filter pushdown for Parquet, adaptive parallel GROUP BY processing, smaller hash-table cells for single-string keys, parallel merge joins, IEJoin for two-inequality conditions, and a cost-based optimizer for distributed plans. The PostgreSQL wire protocol supports psql commands such as \d, \dt, and \dv while the release adds text tokenizers.
One Stone, Three Birds: Finer-Grained Encryption @ Apache Parquet™
Apache Parquet modular encryption is presented as a way to unify fine-grained access restriction, column-specific retention, and encryption at rest for large Hadoop-based data platforms. The approach encrypts selected columns with different keys at the Parquet file-format layer while preserving column projection, predicate pushdown, encoding, and compression; schema-controlled encryption avoids extra RPC calls and supports tag-driven policies. The post describes operational challenges including multiple access paths and Parquet versions, hard versus soft denial, KMS reliability, historical backfills, changing schema tags, and mixed data formats, alongside auto-onboarding and a high-throughput tool for existing data. In an evaluation using Java 8, CTR mode, and 60% encrypted columns, encryption added 5.7% to writes and 3.7% to reads, while the tool encrypted data 20 times faster; the changes were merged upstream for Parquet 1.13.0.
Cost Efficiency @ Scale in Big Data File Format
Uber’s rapid data growth increased storage and access costs, along with hardware demand, resource use, and risks such as out-of-memory conditions and long GC pauses. The work focuses on Parquet-level file-size reduction through ZSTD compression, unused-column pruning, column ordering, decimal precision reduction, and advanced encodings, while benchmarking storage and query costs across representative workloads. Converting top Hive tables from GZIP to ZSTD cut storage by about 7%, and from SNAPPY by about 39%; previously uncompressed tables saw about 79% reduction, while benchmark queries generally used fewer vCore seconds with ZSTD. For offline conversion, level 19 was selected because higher levels did not improve reduced size while increasing write time. The work also reports a 32% reduction from suitable four-column sorting and upstream Parquet changes, including faster translation and high-throughput pruning.
Fast Copy-On-Write within Apache Parquet for Data Lakehouse ACID Upserts
Large-scale lakehouse upserts can become slow and resource-intensive, especially in Apache Hudi copy-on-write mode, where affected files are read, modified, and rewritten in full. The approach adds a Parquet row-level secondary index mapping records to files and row IDs, allowing updates to locate the specific data pages containing changed records. During copy-on-write, related pages are updated while unrelated pages are copied as byte buffers, avoiding decompression, re-encoding, and record reassembly for skipped data. Benchmarks using TPC-DS sales data, Spark jobs with matching vCore and memory settings, and update rates from 5% to 50% found significantly faster speeds than traditional approaches, including Delta Lake’s default configuration. The authors plan to integrate the row-level index and fast copy-on-write into Apache Hudi for use cases such as incremental ingestion, data deletion, and CDC.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.