# How Partition Access Visualizations Reduced our Data Lake S3 Cost by 33%

[Yelp](https://yomu.fyi/company/yelp) · Nick Del Nano, Data Streaming · May 21, 2026

**Type:** Problem & solution

## Summary

Data teams at Yelp struggled to identify table consumers and track how datasets were accessed across their petabyte-scale data lake. To solve this, the team built a usage attribution system using Amazon S3 server access logs to visualize time-based partition keys against access timestamps and IAM roles. These visualizations revealed distinctive patterns for daily batch jobs, backfills, and ad hoc queries, guiding migrations of thousands of tables to Apache Iceberg format. Furthermore, Yelp implemented a Default Access Retention strategy gated by IAM bucket policies to prevent accidental queries from resetting S3 Intelligent Tiering transitions. Combining these granular usage insights with targeted S3 storage classes reduced Yelp's S3 storage costs by 33 percent.

## Context

In large analytics environments, data teams often lack data-driven usage attribution to identify who their stakeholders are and how data is being used. Without clear access patterns, teams rely on outdated documentation and hesitate to commit to deletion-based retention or cold storage policies that carry minimum storage durations and retrieval fees.

## Approach / What changed

Yelp ingested Amazon S3 server access logs into their data lake and used batch SQL transformations to aggregate object GET operations by table, partition value, timestamp, and requester IAM role. They mapped access patterns to distinguish batch jobs, backfills, and ad hoc queries, assigned S3 Intelligent Tiering to datasets with unpredictable patterns, and implemented a Default Access Retention policy gating older partitions behind restrictive bucket IAM policies.

## Takeaways

- Visualizing accessed partition dates against event timestamps reveals clear signatures: diagonal lines indicate daily batch jobs, vertical lines show backfills, and scatter points mark ad hoc queries.
- Under S3 Intelligent Tiering, objects not accessed for 30 days decrease in storage cost by 40%, and objects not accessed for 90 days decrease in storage cost by 81%.
- Default Access Retention uses restrictive IAM bucket policies on older partitions so accidental queries cannot reset S3 Intelligent Tiering tiers or trigger large retrieval fees without an approved Terraform PR.

**Tags:** [Architecture](https://yomu.fyi/topic/architecture), [AWS](https://yomu.fyi/topic/aws), [Data Pipelines](https://yomu.fyi/topic/data-pipelines), [Performance](https://yomu.fyi/topic/performance), [Scalability](https://yomu.fyi/topic/scalability)

[Read original post](https://engineeringblog.yelp.com/2026/05/partition-access-visualizations.html)