Apache DataFusion is an embeddable query engine: SQL goes in, runs as columnar Arrow batches inside your own process, and results come out without a separate service.
Apache DataFusion is a query engine you embed rather than deploy. A SQL statement is parsed and planned, then executed over Arrow columnar batches in the same process as your application, so there is no network hop and no separate cluster to scale. The Cloudflare Logpush team uses exactly this shape: SQL Transformers run per batch inside a Go delivery service, with Arrow IPC carrying typed batches across the language boundary. The trade-off is that the engine lives inside your service, so its memory and batching behavior are yours to manage, and work the engine cannot express as SQL can fall back to slower paths.
Embedding removes a service you would otherwise have to deploy, scale, and maintain. In Cloudflare's Logpush Transformers, SQL runs per batch inside the existing Go delivery service, so there is no extra hop between the data and the query.
Arrow is the columnar format the engine executes over. In the Logpush implementation, Arrow IPC carries typed batches across the language boundary between the Go service and the embedded engine, avoiding row serialization and an added network hop.
UDFs are opaque, so the engine cannot optimize or reorder them the way it can plain SQL. In Spark GPU pipelines, that opacity forces CPU fallback and data transfer overhead; rewriting the logic as SQL expressions or columnar operations is what restores the fast path.
The posts behind it
NVIDIA · · TutorialUser-defined functions (UDFs) in Apache Spark break the declarative abstraction, forcing GPU-accelerated pipelines to fall back to CPU execution and incur data transfer overhead. NVIDIA cuDF addresses this with GPU parallelism, but UDFs remain a bottleneck due to their opaque code. The post describes three approaches to rewrite UDFs into GPU-accelerable forms: SQL expressions, the RapidsUDF interface using cuDF columnar operations, or custom CUDA kernels via JNI. The cuDF skills for Apache Spark automate this process with four agent-guided feedback loops for testing, conversion, benchmarking, and optimization, providing a project template and best practices. A production deployment converted over 30 UDFs, achieving a 22x speedup on a 600-billion-row workload, reducing operator time from 274.55 to 12.78 hours.
Databricks · · AnnouncementLakeflow Designer, a visual no-code data preparation tool in Databricks, added three new operators for cleaning, deduplication, and quality checks, addressing customer requests for more no-code capabilities. The Prepare operator standardizes data by changing case, converting types, replacing values, and applying formulas; Unique removes duplicates across selected columns with ordering control; Guardrails enforces quality rules with configurable failure handling. Existing operators gained multi-table joins with separate outputs for matched, left unmatched, and right unmatched rows, branching in Filter, and dynamic column selection in Select. Output options expanded to materialized views in Unity Catalog that refresh automatically, merge into existing tables, and direct file writes to CSV, Excel, or JSON with dynamic worksheets. Canvas improvements include a redesigned interface with better contrast and larger connection targets for large workflows, a searchable table of contents, visual version history, and live cursor presence.
Cloudflare · · Problem & solutionCloudflare’s Data team manages a pipeline handling over two billion events per second from tens of thousands of servers, powering Logs, SQL API, and observability. To counter exponential traffic growth against linear hardware expansion, they implemented several optimizations over five years, deferring two and a half years of hardware expansion. In 2022, they replaced a monolithic schema with product-specific schemas, reducing backbone bandwidth by 35-40%. In 2024, they switched compression from LZ4 to zstd, achieving a 40% reduction in compressed traffic with negligible CPU impact. In 2026, they decoupled pipeline transit from database redundancy via a shared ingest pool and late replication, applied sampling downstream, and introduced aggregations, eliminating about 70 GB/s of compressed traffic and halving projected 2031 data volume.
Clickhouse · · AnnouncementAt ClickHouse Open House NYC, the team announced WalShadow, an open-source engine that replicates Postgres into ClickHouse from the physical WAL, achieving sub-second latency and high throughput. They also launched ClickHouse Managed Postgres on Google Cloud in private preview, offering Postgres on local NVMe storage with native CDC into ClickHouse. The chdb Postgres extension was released, enabling fast imports from cloud storage using the embeddable ClickHouse engine. A live demo showed WalShadow keeping ClickHouse under 400 ms behind during a simulated trading platform with 50,000 trades per second, and detecting a fraud burst in under 500 ms compared to over 15 seconds with ClickPipes. The integrated stack aims to provide world-class Postgres for transactions and ClickHouse for analytics, with WalShadow as a key step toward unifying OLTP and OLAP.
Cloudflare · · AnnouncementCloudflare Logpush now supports Transformers, allowing teams to filter, redact, and reshape log streams using SQL before export to SIEMs, analytics platforms, or object storage. This eliminates the need for a separate processing service that teams would otherwise have to deploy, scale, and maintain. Transformers are SQL queries applied per batch within Logpush’s high-throughput pipeline, using Apache DataFusion and Arrow for columnar execution. The implementation embeds DataFusion in Bufstream, the Go-based delivery service, using cgo and Arrow IPC to pass typed batches across the language boundary without introducing network hops. Customers can create, preview, and version Transformers via the dashboard or API, and attach them to NDJSON Logpush jobs to reduce downstream data volume and cost.
More like this
A model deployed on AKS is open by default. See how MaaS policies insert Authorino authentication and Limitador rate limits in front of vLLM, so bad keys and over-quota requests stop at the gate.
See how Caddy handles a request for a static site, from DNS resolution through container routing to file_server, and how TLS and redirects fit into the flow.
BYOK lets a system run on assets you supply — your model credential or your private knowledge — instead of the vendor's defaults. This walkthrough traces six steps from registration to an audited response.
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
One useful email. Unsubscribe anytime.
