How Apache DataFusion Runs SQL In-Process
Apache DataFusion is an embeddable query engine: SQL goes in, runs as columnar Arrow batches inside your own process, and results come out without a separate service.
Loading post…
CloudflareChristian EndresProblem & solution
Rewinding the exponential: How we keep hardware usage down
Summary
Cloudflare’s Data team manages a pipeline handling over two billion events per second from tens of thousands of servers, powering Logs, SQL API, and observability. To counter exponential traffic growth against linear hardware expansion, they implemented several optimizations over five years, deferring two and a half years of hardware expansion. In 2022, they replaced a monolithic schema with product-specific schemas, reducing backbone bandwidth by 35-40%. In 2024, they switched compression from LZ4 to zstd, achieving a 40% reduction in compressed traffic with negligible CPU impact. In 2026, they decoupled pipeline transit from database redundancy via a shared ingest pool and late replication, applied sampling downstream, and introduced aggregations, eliminating about 70 GB/s of compressed traffic and halving projected 2031 data volume.
Context
Cloudflare’s Data team manages the data pipeline that handles over two billion events per second from tens of thousands of servers across the global network, supporting products like Cloudflare Logs, SQL API, and observability.
Approach / What changed
The team implemented a series of optimizations: breaking the monolithic schema into product-specific schemas, switching compression from LZ4 to zstd, decoupling pipeline transit from database redundancy using a shared ingest pool and late replication, applying sampling downstream, and introducing aggregations into the pipeline.
Takeaways
Replacing a monolithic schema with product-specific schemas reduced backbone bandwidth by 35-40%, deferring about a year of traffic growth.
Switching from LZ4 to zstd compression achieved a 40% reduction in compressed traffic with negligible CPU impact.
Decoupling pipeline transit from database redundancy and applying sampling downstream eliminated about 70 GB/s of compressed traffic and halved projected 2031 data volume.
FAQ
How did Cloudflare reduce bandwidth without increasing CPU?
They switched from LZ4 to zstd compression, which provided a 40% reduction in compressed traffic with negligible CPU impact after careful rollout and monitoring.
What was the key change to avoid double-writing for disaster resilience?
They decoupled pipeline transit from database redundancy by sending each event once to a shared ingest pool and using late replication from centralized regions, eliminating the need for Logfwdr to send two copies.
How did sampling help reduce data volume?
Sampling was applied downstream after centralization, allowing high-volume traffic to be downsampled while rare events are preserved, reducing compressed traffic by about 25 GB/s across top datasets.
Source and freshness
- Source
- blog.cloudflare.com
- Published
- Added to Yomu
Summaries are generated from the original by a language model. Check the original before relying on it.
See the idea
Visual explainers
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
The best engineering reads, once a week
One useful email. Unsubscribe anytime.