---
title: "Rewinding the exponential: How we keep hardware usage down"
description: "Cloudflare’s Data team manages a pipeline handling over two billion events per second from tens of thousands of servers, powering Logs, SQL API, and observability. To counter exponential traffic growth against linear hardware expansion, they implemented several optimizations over five years, deferring two and a half years of hardware expansion. In 2022, they replaced a monolithic schema with product-specific schemas, reducing backbone bandwidth by 35-40%. In 2024, they switched compression from LZ4 to zstd, achieving a 40% reduction in compressed traffic with negligible CPU impact. In 2026, they decoupled pipeline transit from database redundancy via a shared ingest pool and late replication, applied sampling downstream, and introduced aggregations, eliminating about 70 GB/s of compressed traffic and halving projected 2031 data volume."
---

# Rewinding the exponential: How we keep hardware usage down

[Cloudflare](https://yomu.fyi/company/cloudflare) · Christian Endres · Oct 13, 2026

**Type:** Problem & solution

## Summary

Cloudflare’s Data team manages a pipeline handling over two billion events per second from tens of thousands of servers, powering Logs, SQL API, and observability. To counter exponential traffic growth against linear hardware expansion, they implemented several optimizations over five years, deferring two and a half years of hardware expansion. In 2022, they replaced a monolithic schema with product-specific schemas, reducing backbone bandwidth by 35-40%. In 2024, they switched compression from LZ4 to zstd, achieving a 40% reduction in compressed traffic with negligible CPU impact. In 2026, they decoupled pipeline transit from database redundancy via a shared ingest pool and late replication, applied sampling downstream, and introduced aggregations, eliminating about 70 GB/s of compressed traffic and halving projected 2031 data volume.

## Context

Cloudflare’s Data team manages the data pipeline that handles over two billion events per second from tens of thousands of servers across the global network, supporting products like Cloudflare Logs, SQL API, and observability.

## Approach / What changed

The team implemented a series of optimizations: breaking the monolithic schema into product-specific schemas, switching compression from LZ4 to zstd, decoupling pipeline transit from database redundancy using a shared ingest pool and late replication, applying sampling downstream, and introducing aggregations into the pipeline.

## Takeaways

- Replacing a monolithic schema with product-specific schemas reduced backbone bandwidth by 35-40%, deferring about a year of traffic growth.
- Switching from LZ4 to zstd compression achieved a 40% reduction in compressed traffic with negligible CPU impact.
- Decoupling pipeline transit from database redundancy and applying sampling downstream eliminated about 70 GB/s of compressed traffic and halved projected 2031 data volume.

## FAQ

### How did Cloudflare reduce bandwidth without increasing CPU?

They switched from LZ4 to zstd compression, which provided a 40% reduction in compressed traffic with negligible CPU impact after careful rollout and monitoring.

### What was the key change to avoid double-writing for disaster resilience?

They decoupled pipeline transit from database redundancy by sending each event once to a shared ingest pool and using late replication from centralized regions, eliminating the need for Logfwdr to send two copies.

### How did sampling help reduce data volume?

Sampling was applied downstream after centralization, allowing high-volume traffic to be downsampled while rare events are preserved, reducing compressed traffic by about 25 GB/s across top datasets.

**Tags:** [Cloudflare](https://yomu.fyi/topic/cloudflare), [Compression](https://yomu.fyi/topic/compression), [Data Engineering](https://yomu.fyi/topic/data-engineering), [Infrastructure](https://yomu.fyi/topic/infrastructure), [Sampling](https://yomu.fyi/topic/sampling)

- Source: [Cloudflare](https://blog.cloudflare.com/rewinding-the-exponential)
- Source URL: https://blog.cloudflare.com/rewinding-the-exponential
- Ingested by Yomu: 2026-10-13T18:00:15.158Z

[Read original post](https://blog.cloudflare.com/rewinding-the-exponential)