Topic

Compression

2 posts about Compression. Every summary links to the original.

Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.

Newest first

Posts

Cloudflare
Rewinding the exponential: How we keep hardware usage down
Cloudflare’s Data team manages a pipeline handling over two billion events per second from tens of thousands of servers, powering Logs, SQL API, and observability. To counter exponential traffic growth against linear hardware expansion, they implemented several optimizations over five years, deferring two and a half years of hardware expansion. In 2022, they replaced a monolithic schema with product-specific schemas, reducing backbone bandwidth by 35-40%. In 2024, they switched compression from LZ4 to zstd, achieving a 40% reduction in compressed traffic with negligible CPU impact. In 2026, they decoupled pipeline transit from database redundancy via a shared ingest pool and late replication, applied sampling downstream, and introduced aggregations, eliminating about 70 GB/s of compressed traffic and halving projected 2031 data volume.

Uber
Cost Efficiency @ Scale in Big Data File Format
Uber’s rapid data growth increased storage and access costs, along with hardware demand, resource use, and risks such as out-of-memory conditions and long GC pauses. The work focuses on Parquet-level file-size reduction through ZSTD compression, unused-column pruning, column ordering, decimal precision reduction, and advanced encodings, while benchmarking storage and query costs across representative workloads. Converting top Hive tables from GZIP to ZSTD cut storage by about 7%, and from SNAPPY by about 39%; previously uncompressed tables saw about 79% reduction, while benchmark queries generally used fewer vCore seconds with ZSTD. For offline conversion, level 19 was selected because higher levels did not improve reduced size while increasing write time. The work also reports a 32% reduction from suitable four-column sorting and upstream Parquet changes, including faster translation and high-throughput pruning.

The best engineering reads, once a week.

Topic

2 posts about Compression. Every summary links to the original.

Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.

Newest first

Cloudflare
Rewinding the exponential: How we keep hardware usage down
Cloudflare’s Data team manages a pipeline handling over two billion events per second from tens of thousands of servers, powering Logs, SQL API, and observability. To counter exponential traffic growth against linear hardware expansion, they implemented several optimizations over five years, deferring two and a half years of hardware expansion. In 2022, they replaced a monolithic schema with product-specific schemas, reducing backbone bandwidth by 35-40%. In 2024, they switched compression from LZ4 to zstd, achieving a 40% reduction in compressed traffic with negligible CPU impact. In 2026, they decoupled pipeline transit from database redundancy via a shared ingest pool and late replication, applied sampling downstream, and introduced aggregations, eliminating about 70 GB/s of compressed traffic and halving projected 2031 data volume.

Uber
Cost Efficiency @ Scale in Big Data File Format
Uber’s rapid data growth increased storage and access costs, along with hardware demand, resource use, and risks such as out-of-memory conditions and long GC pauses. The work focuses on Parquet-level file-size reduction through ZSTD compression, unused-column pruning, column ordering, decimal precision reduction, and advanced encodings, while benchmarking storage and query costs across representative workloads. Converting top Hive tables from GZIP to ZSTD cut storage by about 7%, and from SNAPPY by about 39%; previously uncompressed tables saw about 79% reduction, while benchmark queries generally used fewer vCore seconds with ZSTD. For offline conversion, level 19 was selected because higher levels did not improve reduced size while increasing write time. The work also reports a 32% reduction from suitable four-column sorting and upstream Parquet changes, including faster translation and high-throughput pruning.

The best engineering reads, once a week.