How Apache DataFusion Runs SQL In-Process
Apache DataFusion is an embeddable query engine: SQL goes in, runs as columnar Arrow batches inside your own process, and results come out without a separate service.
Loading…
Topic
Apache DataFusion
1 post about Apache DataFusion. Every summary links to the original.
- 1 post
- 1 company
- 1 explainer
See the idea
Apache DataFusion explainers
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
Filter, reshape, and redact logs before export with Transformers
Cloudflare Logpush now supports Transformers, allowing teams to filter, redact, and reshape log streams using SQL before export to SIEMs, analytics platforms, or object storage. This eliminates the need for a separate processing service that teams would otherwise have to deploy, scale, and maintain. Transformers are SQL queries applied per batch within Logpush’s high-throughput pipeline, using Apache DataFusion and Arrow for columnar execution. The implementation embeds DataFusion in Bufstream, the Go-based delivery service, using cgo and Arrow IPC to pass typed batches across the language boundary without introducing network hops. Customers can create, preview, and version Transformers via the dashboard or API, and attach them to NDJSON Logpush jobs to reduce downstream data volume and cost.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.