---
title: "How Apache DataFusion Runs SQL In-Process"
description: "Apache DataFusion is an embeddable query engine: SQL goes in, runs as columnar Arrow batches inside your own process, and results come out without a separate service."
---

# How Apache DataFusion Runs SQL In-Process

**Type:** [How it works](https://yomu.fyi/visual-explainers/category/how-it-works.md)

![A left-to-right diagram of an embedded Apache DataFusion engine inside an application process boundary, showing four numbered stages: SQL query, plan, columnar execution over Arrow batches, and results, plus a strip showing Arrow IPC between a Go service and the engine with no network hop.](https://yomufyi.s3.us-east-1.amazonaws.com/visual-explainers/b818a479-f2d8-4d40-93bd-e8de402f9c2b/image-dc1830d9-fff4-4e59-b1ae-6b6dfafde64d.webp)

> Apache DataFusion is an embeddable query engine: SQL goes in, runs as columnar Arrow batches inside your own process, and results come out without a separate service.

## Explanation

Apache DataFusion is a query engine you embed rather than deploy. A SQL statement is parsed and planned, then executed over Arrow columnar batches in the same process as your application, so there is no network hop and no separate cluster to scale. The Cloudflare Logpush team uses exactly this shape: SQL Transformers run per batch inside a Go delivery service, with Arrow IPC carrying typed batches across the language boundary. The trade-off is that the engine lives inside your service, so its memory and batching behavior are yours to manage, and work the engine cannot express as SQL can fall back to slower paths.

## FAQ

### Why embed a query engine instead of running a separate processing service?

Embedding removes a service you would otherwise have to deploy, scale, and maintain. In Cloudflare's Logpush Transformers, SQL runs per batch inside the existing Go delivery service, so there is no extra hop between the data and the query.

### What role does Apache Arrow play in a DataFusion pipeline?

Arrow is the columnar format the engine executes over. In the Logpush implementation, Arrow IPC carries typed batches across the language boundary between the Go service and the embedded engine, avoiding row serialization and an added network hop.

### Where does a user-defined function hurt a declarative query engine?

UDFs are opaque, so the engine cannot optimize or reorder them the way it can plain SQL. In Spark GPU pipelines, that opacity forces CPU fallback and data transfer overhead; rewriting the logic as SQL expressions or columnar operations is what restores the fast path.

**Tags:** [Apache Arrow](https://yomu.fyi/topic/apache-arrow.md), [Apache DataFusion](https://yomu.fyi/topic/apache-datafusion.md), [Data Engineering](https://yomu.fyi/topic/data-engineering.md), [Parquet](https://yomu.fyi/topic/parquet.md), [Rust](https://yomu.fyi/topic/rust.md), [SQL](https://yomu.fyi/topic/sql.md), [UDF](https://yomu.fyi/topic/udf.md)

## Related reading

### [How to Rewrite Apache Spark UDFs for GPU Acceleration with AI Agents](https://yomu.fyi/post/how-to-rewrite-apache-spark-udfs-for-gpu-acceleration-with-ai-agents.md)
- Company: [NVIDIA](https://yomu.fyi/company/nvidia-developer-blog.md)
- Author: Elizabeth Goodman
- Published: Oct 13, 2026
- Type: Tutorial

User-defined functions (UDFs) in Apache Spark break the declarative abstraction, forcing GPU-accelerated pipelines to fall back to CPU execution and incur data transfer overhead. NVIDIA cuDF addresses this with GPU parallelism, but UDFs remain a bottleneck due to their opaque code. The post describes three approaches to rewrite UDFs into GPU-accelerable forms: SQL expressions, the RapidsUDF interface using cuDF columnar operations, or custom CUDA kernels via JNI. The cuDF skills for Apache Spark automate this process with four agent-guided feedback loops for testing, conversion, benchmarking, and optimization, providing a project template and best practices. A production deployment converted over 30 UDFs, achieving a 22x speedup on a 600-billion-row workload, reducing operator time from 274.55 to 12.78 hours.


### [What's New in Lakeflow Designer - October 2026 Roundup](https://yomu.fyi/post/what-s-new-in-lakeflow-designer-october-2026-roundup.md)
- Company: [Databricks](https://yomu.fyi/company/databricks.md)
- Author: Jason Messer, Emanuel Zgraggen, V Maharajh, Kyrylo Bakumenko, Xiao Yan, Matt Jones
- Published: Oct 13, 2026
- Type: Announcement

Lakeflow Designer, a visual no-code data preparation tool in Databricks, added three new operators for cleaning, deduplication, and quality checks, addressing customer requests for more no-code capabilities. The Prepare operator standardizes data by changing case, converting types, replacing values, and applying formulas; Unique removes duplicates across selected columns with ordering control; Guardrails enforces quality rules with configurable failure handling. Existing operators gained multi-table joins with separate outputs for matched, left unmatched, and right unmatched rows, branching in Filter, and dynamic column selection in Select. Output options expanded to materialized views in Unity Catalog that refresh automatically, merge into existing tables, and direct file writes to CSV, Excel, or JSON with dynamic worksheets. Canvas improvements include a redesigned interface with better contrast and larger connection targets for large workflows, a searchable table of contents, visual version history, and live cursor presence.


### [Rewinding the exponential: How we keep hardware usage down](https://yomu.fyi/post/rewinding-the-exponential-how-we-keep-hardware-usage-down.md)
- Company: [Cloudflare](https://yomu.fyi/company/cloudflare.md)
- Author: Christian Endres
- Published: Oct 13, 2026
- Type: Problem & solution

Cloudflare’s Data team manages a pipeline handling over two billion events per second from tens of thousands of servers, powering Logs, SQL API, and observability. To counter exponential traffic growth against linear hardware expansion, they implemented several optimizations over five years, deferring two and a half years of hardware expansion. In 2022, they replaced a monolithic schema with product-specific schemas, reducing backbone bandwidth by 35-40%. In 2024, they switched compression from LZ4 to zstd, achieving a 40% reduction in compressed traffic with negligible CPU impact. In 2026, they decoupled pipeline transit from database redundancy via a shared ingest pool and late replication, applied sampling downstream, and introduced aggregations, eliminating about 70 GB/s of compressed traffic and halving projected 2031 data volume.


### [Postgres at Open House NYC](https://yomu.fyi/post/postgres-at-open-house-nyc.md)
- Company: [Clickhouse](https://yomu.fyi/company/clickhouse.md)
- Published: Oct 13, 2026
- Type: Announcement

At ClickHouse Open House NYC, the team announced WalShadow, an open-source engine that replicates Postgres into ClickHouse from the physical WAL, achieving sub-second latency and high throughput. They also launched ClickHouse Managed Postgres on Google Cloud in private preview, offering Postgres on local NVMe storage with native CDC into ClickHouse. The chdb Postgres extension was released, enabling fast imports from cloud storage using the embeddable ClickHouse engine. A live demo showed WalShadow keeping ClickHouse under 400 ms behind during a simulated trading platform with 50,000 trades per second, and detecting a fraud burst in under 500 ms compared to over 15 seconds with ClickPipes. The integrated stack aims to provide world-class Postgres for transactions and ClickHouse for analytics, with WalShadow as a key step toward unifying OLTP and OLAP.


### [Filter, reshape, and redact logs before export with Transformers](https://yomu.fyi/post/filter-reshape-and-redact-logs-before-export-with-transformers.md)
- Company: [Cloudflare](https://yomu.fyi/company/cloudflare.md)
- Author: Sahidya Devadoss
- Published: Oct 12, 2026
- Type: Announcement

Cloudflare Logpush now supports Transformers, allowing teams to filter, redact, and reshape log streams using SQL before export to SIEMs, analytics platforms, or object storage. This eliminates the need for a separate processing service that teams would otherwise have to deploy, scale, and maintain. Transformers are SQL queries applied per batch within Logpush’s high-throughput pipeline, using Apache DataFusion and Arrow for columnar execution. The implementation embeds DataFusion in Bufstream, the Go-based delivery service, using cgo and Arrow IPC to pass typed batches across the language boundary without introducing network hops. Customers can create, preview, and version Transformers via the dashboard or API, and attach them to NDJSON Logpush jobs to reduce downstream data volume and cost.