How Partition Access Visualizations Reduced our Data Lake S3 Cost by 33%

Summary

Data teams at Yelp struggled to identify table consumers and track how datasets were accessed across their petabyte-scale data lake. To solve this, the team built a usage attribution system using Amazon S3 server access logs to visualize time-based partition keys against access timestamps and IAM roles. These visualizations revealed distinctive patterns for daily batch jobs, backfills, and ad hoc queries, guiding migrations of thousands of tables to Apache Iceberg format. Furthermore, Yelp implemented a Default Access Retention strategy gated by IAM bucket policies to prevent accidental queries from resetting S3 Intelligent Tiering transitions. Combining these granular usage insights with targeted S3 storage classes reduced Yelp's S3 storage costs by 33 percent.

Takeaways

1. 1

   Visualizing accessed partition dates against event timestamps reveals clear signatures: diagonal lines indicate daily batch jobs, vertical lines show backfills, and scatter points mark ad hoc queries.

2. 2

   Under S3 Intelligent Tiering, objects not accessed for 30 days decrease in storage cost by 40%, and objects not accessed for 90 days decrease in storage cost by 81%.

3. 3

   Default Access Retention uses restrictive IAM bucket policies on older partitions so accidental queries cannot reset S3 Intelligent Tiering tiers or trigger large retrieval fees without an approved Terraform PR.