---
title: "Cost Efficiency @ Scale in Big Data File Format"
description: "Uber’s rapid data growth increased storage and access costs, along with hardware demand, resource use, and risks such as out-of-memory conditions and long GC pauses. The work focuses on Parquet-level file-size reduction through ZSTD compression, unused-column pruning, column ordering, decimal precision reduction, and advanced encodings, while benchmarking storage and query costs across representative workloads. Converting top Hive tables from GZIP to ZSTD cut storage by about 7%, and from SNAPPY by about 39%; previously uncompressed tables saw about 79% reduction, while benchmark queries generally used fewer vCore seconds with ZSTD. For offline conversion, level 19 was selected because higher levels did not improve reduced size while increasing write time. The work also reports a 32% reduction from suitable four-column sorting and upstream Parquet changes, including faster translation and high-throughput pruning."
---

# Cost Efficiency @ Scale in Big Data File Format

[Uber](https://yomu.fyi/company/uber) · May 19, 2026

**Type:** Problem & solution

## Summary

Uber’s rapid data growth increased storage and access costs, along with hardware demand, resource use, and risks such as out-of-memory conditions and long GC pauses. The work focuses on Parquet-level file-size reduction through ZSTD compression, unused-column pruning, column ordering, decimal precision reduction, and advanced encodings, while benchmarking storage and query costs across representative workloads. Converting top Hive tables from GZIP to ZSTD cut storage by about 7%, and from SNAPPY by about 39%; previously uncompressed tables saw about 79% reduction, while benchmark queries generally used fewer vCore seconds with ZSTD. For offline conversion, level 19 was selected because higher levels did not improve reduced size while increasing write time. The work also reports a 32% reduction from suitable four-column sorting and upstream Parquet changes, including faster translation and high-throughput pruning.

## Context

Uber’s growth increased data volume, access loads, storage and compute costs, hardware requirements, resource usage, and risks including out-of-memory conditions and high garbage-collection pauses. The work targets storage cost efficiency at the file-format level, primarily for Parquet data in Uber’s data lake.

## Approach / What changed

The initiatives optimize Parquet through ZSTD compression, compression-level selection, unused-column pruning based on query-engine audit logs, column ordering, decimal precision reduction, and advanced encodings. The work also adds Parquet tooling and upstream changes for faster large-scale translation and pruning, supported by storage and query-performance benchmarks.

## Takeaways

- ZSTD reduced storage by about 7% versus GZIP and about 39% versus SNAPPY in the selected top 10 Hive tables; some previously uncompressed tables saw about 79% reduction.
- ZSTD decompression time stayed constant across tested compression levels, while write time increased with higher levels; level 19 was chosen for offline translation because higher levels did not improve reduced size.
- Sorting UUID, time\_ms, latitude, and longitude in the tested order reduced data size by 32% versus no sorting for one table and partition.

**Tags:** [Compression](https://yomu.fyi/topic/compression), [Parquet](https://yomu.fyi/topic/parquet)

- Source: [Uber](https://www.uber.com/us/en/blog/cost-efficiency-big-data)
- Source URL: https://www.uber.com/us/en/blog/cost-efficiency-big-data
- Ingested by Yomu: 2026-09-13T04:02:08.665Z

[Read original post](https://www.uber.com/us/en/blog/cost-efficiency-big-data)