---
title: "Compression"
description: "2 posts about Compression, summarised, each linking to the original."
---

# Compression
> 2 posts about Compression, summarised, each linking to the original.

## Articles

### [Rewinding the exponential: How we keep hardware usage down](https://yomu.fyi/post/rewinding-the-exponential-how-we-keep-hardware-usage-down.md)
- Company: [Cloudflare](https://yomu.fyi/company/cloudflare.md)
- Author: Christian Endres
- Published: Oct 13, 2026
- Type: Problem & solution

Cloudflare’s Data team manages a pipeline handling over two billion events per second from tens of thousands of servers, powering Logs, SQL API, and observability. To counter exponential traffic growth against linear hardware expansion, they implemented several optimizations over five years, deferring two and a half years of hardware expansion. In 2022, they replaced a monolithic schema with product-specific schemas, reducing backbone bandwidth by 35-40%. In 2024, they switched compression from LZ4 to zstd, achieving a 40% reduction in compressed traffic with negligible CPU impact. In 2026, they decoupled pipeline transit from database redundancy via a shared ingest pool and late replication, applied sampling downstream, and introduced aggregations, eliminating about 70 GB/s of compressed traffic and halving projected 2031 data volume.


### [Cost Efficiency @ Scale in Big Data File Format](https://yomu.fyi/post/cost-efficiency-scale-in-big-data-file-format.md)
- Company: [Uber](https://yomu.fyi/company/uber.md)
- Published: May 19, 2026
- Type: Problem & solution

Uber’s rapid data growth increased storage and access costs, along with hardware demand, resource use, and risks such as out-of-memory conditions and long GC pauses. The work focuses on Parquet-level file-size reduction through ZSTD compression, unused-column pruning, column ordering, decimal precision reduction, and advanced encodings, while benchmarking storage and query costs across representative workloads. Converting top Hive tables from GZIP to ZSTD cut storage by about 7%, and from SNAPPY by about 39%; previously uncompressed tables saw about 79% reduction, while benchmark queries generally used fewer vCore seconds with ZSTD. For offline conversion, level 19 was selected because higher levels did not improve reduced size while increasing write time. The work also reports a 32% reduction from suitable four-column sorting and upstream Parquet changes, including faster translation and high-throughput pruning.