Topic

Sampling

1 post about Sampling. Every summary links to the original.

Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.

Newest first

Posts

Cloudflare
Rewinding the exponential: How we keep hardware usage down
Cloudflare’s Data team manages a pipeline handling over two billion events per second from tens of thousands of servers, powering Logs, SQL API, and observability. To counter exponential traffic growth against linear hardware expansion, they implemented several optimizations over five years, deferring two and a half years of hardware expansion. In 2022, they replaced a monolithic schema with product-specific schemas, reducing backbone bandwidth by 35-40%. In 2024, they switched compression from LZ4 to zstd, achieving a 40% reduction in compressed traffic with negligible CPU impact. In 2026, they decoupled pipeline transit from database redundancy via a shared ingest pool and late replication, applied sampling downstream, and introduced aggregations, eliminating about 70 GB/s of compressed traffic and halving projected 2031 data volume.

The best engineering reads, once a week.

Topic

1 post about Sampling. Every summary links to the original.

Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.

Newest first

Cloudflare
Rewinding the exponential: How we keep hardware usage down
Cloudflare’s Data team manages a pipeline handling over two billion events per second from tens of thousands of servers, powering Logs, SQL API, and observability. To counter exponential traffic growth against linear hardware expansion, they implemented several optimizations over five years, deferring two and a half years of hardware expansion. In 2022, they replaced a monolithic schema with product-specific schemas, reducing backbone bandwidth by 35-40%. In 2024, they switched compression from LZ4 to zstd, achieving a 40% reduction in compressed traffic with negligible CPU impact. In 2026, they decoupled pipeline transit from database redundancy via a shared ingest pool and late replication, applied sampling downstream, and introduced aggregations, eliminating about 70 GB/s of compressed traffic and halving projected 2031 data volume.

The best engineering reads, once a week.