---
title: "Running Cost-Efficient Export Workloads at Uber"
description: "Uber’s offline export workloads often retrieve a small set of records from large, date-partitioned Apache Hudi tables, but narrow historical predicates can still trigger full-table scans. Repeatedly touching files, metadata, or Parquet footers keeps Google Cloud Storage objects hot, undermining Auto-Class tiering and increasing storage, retrieval, metadata-operation, egress, and latency costs. The approach combines Hudi column stats with sorting on predicate columns: metadata-table statistics prune files without opening them, while clustered records tighten file-level min/max ranges and make pruning more selective. A Hudi secondary index was evaluated but did not reduce file scans enough because records for a subject were spread across most files. The conclusion is that a one-time rewrite and ongoing Hudi table services can turn broad historical scans into lower-touch lookups, while future work targets multi-column layouts and automated ROI-based optimization."
---

# Running Cost-Efficient Export Workloads at Uber

[Uber](https://yomu.fyi/company/uber) · Aug 12, 2026

**Type:** Problem & solution

## Summary

Uber’s offline export workloads often retrieve a small set of records from large, date-partitioned Apache Hudi tables, but narrow historical predicates can still trigger full-table scans. Repeatedly touching files, metadata, or Parquet footers keeps Google Cloud Storage objects hot, undermining Auto-Class tiering and increasing storage, retrieval, metadata-operation, egress, and latency costs. The approach combines Hudi column stats with sorting on predicate columns: metadata-table statistics prune files without opening them, while clustered records tighten file-level min/max ranges and make pruning more selective. A Hudi secondary index was evaluated but did not reduce file scans enough because records for a subject were spread across most files. The conclusion is that a one-time rewrite and ongoing Hudi table services can turn broad historical scans into lower-touch lookups, while future work targets multi-column layouts and automated ROI-based optimization.

## Context

Selective export queries, including compliance and privacy flows such as DSARs, return small record sets while searching across large historical datasets. Because these queries can touch many partitions, files, metadata entries, and Parquet footers, they keep Google Cloud Storage data hot, prevent effective Auto-Class tiering, and increase storage, retrieval, metadata-operation, egress, and latency costs.

## Approach / What changed

The solution combines Hudi column statistics with sorting or clustering on predicate columns. Hudi’s metadata table stores file-level statistics such as min, max, null count, and total count, allowing irrelevant files to be pruned without opening the underlying Parquet files. Sorting groups matching records together, tightens min/max ranges, and makes pruning more selective. Hudi Table Services manage sorting, index construction, backfills, and incremental sorting.

## Takeaways

- Hudi column stats enable file pruning through the metadata table, avoiding repeated Parquet footer reads that can keep cold Google Cloud Storage files hot.
- Sorting records by subject-oriented predicate columns clusters matches into fewer files and improves the effectiveness of min/max-based pruning.
- A secondary index did not sufficiently reduce scan and metadata costs because records for the same subject were distributed across most files.

**Tags:** [Cost Optimization](https://yomu.fyi/topic/cost-optimization), [Google Cloud](https://yomu.fyi/topic/gcp), [Object Storage](https://yomu.fyi/topic/object-storage)

- Source: [Uber](https://www.uber.com/us/en/blog/running-cost-efficient-export)
- Source URL: https://www.uber.com/us/en/blog/running-cost-efficient-export
- Ingested by Yomu: 2026-09-10T00:02:02.359Z

[Read original post](https://www.uber.com/us/en/blog/running-cost-efficient-export)