How Apache DataFusion Runs SQL In-Process
Apache DataFusion is an embeddable query engine: SQL goes in, runs as columnar Arrow batches inside your own process, and results come out without a separate service.
Loading post…
CloudflareSahidya DevadossAnnouncement
Filter, reshape, and redact logs before export with Transformers
Summary
Cloudflare Logpush now supports Transformers, allowing teams to filter, redact, and reshape log streams using SQL before export to SIEMs, analytics platforms, or object storage. This eliminates the need for a separate processing service that teams would otherwise have to deploy, scale, and maintain. Transformers are SQL queries applied per batch within Logpush’s high-throughput pipeline, using Apache DataFusion and Arrow for columnar execution. The implementation embeds DataFusion in Bufstream, the Go-based delivery service, using cgo and Arrow IPC to pass typed batches across the language boundary without introducing network hops. Customers can create, preview, and version Transformers via the dashboard or API, and attach them to NDJSON Logpush jobs to reduce downstream data volume and cost.
Context
Cloudflare Logpush now includes Transformers, a SQL-based feature that allows teams to filter, redact, and reshape log records before export, eliminating the need for separate processing services.
Approach / What changed
Embedding Apache DataFusion, a Rust SQL engine, into Bufstream (the Go-based Logpush delivery service) using cgo and Arrow IPC to execute SQL transformations in-process without network hops.
Takeaways
Transformers use SQL queries to filter, redact, and reshape logs before export, reducing downstream data volume and cost.
The implementation embeds Apache DataFusion in Bufstream using cgo and Arrow IPC, avoiding network calls and maintaining high throughput.
Customers can create, preview, and version Transformers via the Cloudflare dashboard or API, and attach them to NDJSON Logpush jobs.
FAQ
How do Transformers work?
A Transformer is a SQL query written for a specific Logpush dataset. Logpush applies the query to each batch before export, removing records that do not match the WHERE clause and exporting the output of the SELECT statement in NDJSON format.
How is the SQL executed?
Bufstream executes the SQL query using Apache DataFusion embedded in-process. It passes typed Arrow batches across the Go-Rust boundary via cgo and Arrow IPC, avoiding network calls and maintaining high throughput.
What happens if a transform fails?
If a transform fails, Bufstream does not advance the delivery offset or send untransformed data. It retries with progressive delays and eventually disables the job if it remains in a failing state.
Source and freshness
- Source
- blog.cloudflare.com
- Published
- Added to Yomu
Summaries are generated from the original by a language model. Check the original before relying on it.
See the idea
Visual explainers
APAC Is Not One Region: The Regionalize vs. Standardize Tradeoff
APAC is a federation of markets, not a single deployment region. This diagram maps the tradeoff between local autonomy and one shared platform, and the conditions that decide it.
The AMR Surveillance Stack: From Raw Reads to Governed Answers
Antimicrobial resistance surveillance is a layered data architecture: raw reads and demand signals flow in, get normalized into governed tables, and answer plain-language queries. The diagram maps each boundary.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
The best engineering reads, once a week
One useful email. Unsubscribe anytime.