How Apache DataFusion Runs SQL In-Process
Apache DataFusion is an embeddable query engine: SQL goes in, runs as columnar Arrow batches inside your own process, and results come out without a separate service.
Loading…
Topic
Apache Arrow
5 posts about Apache Arrow. Every summary links to the original.
- 5 posts
- 4 companies
- 2 explainers
See the idea
Apache Arrow explainers
How Apache Arrow Moves Data Without Copies
Apache Arrow is a columnar in-memory format that lets databases, analytics engines, and Python libraries share data without serialization overhead or format conversion, cutting memory peaks and speeding up common workflows.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
Filter, reshape, and redact logs before export with Transformers
Cloudflare Logpush now supports Transformers, allowing teams to filter, redact, and reshape log streams using SQL before export to SIEMs, analytics platforms, or object storage. This eliminates the need for a separate processing service that teams would otherwise have to deploy, scale, and maintain. Transformers are SQL queries applied per batch within Logpush’s high-throughput pipeline, using Apache DataFusion and Arrow for columnar execution. The implementation embeds DataFusion in Bufstream, the Go-based delivery service, using cgo and Arrow IPC to pass typed batches across the language boundary without introducing network hops. Customers can create, preview, and version Transformers via the dashboard or API, and attach them to NDJSON Logpush jobs to reduce downstream data volume and cost.
How iSAM Funds built an options research platform for a 10,000x data problem with ClickHouse Cloud
iSAM Funds, a UK-based systematic asset manager, built an options research platform on ClickHouse Cloud to handle the 10,000x data fan-out from underlying instruments to related options. Migrating from Postgres, the team achieved 35-50x compression and increased single-core ingestion from 8,000 to 800,000 rows per second, while ClickHouse’s native Apache Arrow support eliminated encoding reshuffling. ClickHouse’s columnar storage and vectorized execution made queries fast enough to run on demand, allowing the team to drop pre-computed rollups and materialized views and remove a layer of pipeline complexity. The managed service integrates with iSAM’s AWS footprint and enterprise authentication, pushes backups to iSAM’s own buckets for developer use, and lets them scale instances up for large backfills then down to control costs. The platform gives researchers fast, interactive access to full options history, shortening the time from hypothesis to tested strategy.
ClickHouse is now available on the dbt platform
The ClickHouse adapter for dbt v2, dbt’s Rust rewrite, entered public beta on September 16, 2026, while ClickHouse became a natively supported data warehouse on the dbt platform in private beta. The v2 adapter ships inside the open-source dbt binary, connects through an ADBC driver using Apache Arrow, and supports both open-source ClickHouse and ClickHouse Cloud. Existing models, tests, and profiles carry over, although some features remain incomplete and dbt plans to support the v1 and v2 adapters in parallel. The platform integration provides ClickHouse connections for Studio, environments, scheduled jobs, and Catalog, while the planned path to general availability includes parity items such as ON CLUSTER DDL and distributed materializations, followed by Fusion capabilities.
Apache Arrow Cut Snowflake Fetch Memory Growth by Up to 79%
An ML training run was exhausting worker memory before training could start because a cold Snowflake fetch peaked at 88.8 GiB RSS, versus 17.6 GiB when loading local Parquet. The shared Snowflake-to-pandas path created Python objects and SQLAlchemy Row proxies before pandas built the DataFrame, with temporary representations requiring roughly three times the final DataFrame size at peak. An Apache Arrow path for supported reads reduced median peak process RSS growth by 60–79% across three query shapes, while peak Python allocations fell by 69–87%. The narrow and wide tests also ran faster, but aggregate timing was mixed, so elapsed time was not the headline claim. Existing jobs adopted the default without query rewrites, with normalization preserving the pandas contract and SQLAlchemy retained for unsupported reads; in one workflow, lower fetch peaks doubled the training-data window on the same cluster size.
Introducing Arrow UDFs in PySpark: A Faster, Leaner Replacement for Pandas UDFs
Apache Spark’s Pandas UDFs improve Python UDF throughput through Arrow-based serialization and batching, but Pandas/Arrow conversion can add copies, limit zero-copy behavior, and weaken support for complex datatypes. Native Arrow UDFs, introduced in Databricks Runtime 18.0, operate directly on Arrow data without converting inputs to Pandas or NumPy objects, preserving columnar layout and enabling vectorized processing through Arrow’s compute and memory model. The new @arrow_udf decorator supports scalar, aggregate, and table functions, while type-hinted @udf definitions, mapInArrow, and applyInArrow extend the interface to DataFrame and grouped operations; iterator-based modes can amortize initialization costs. In a comparison, an Arrow UDF ran about 10% faster than a Pandas UDF and used about 40% less memory, with better complex-datatype support.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.