How Partition Access Visualizations Reduced our Data Lake S3 Cost by 33%

Summary

Data teams at Yelp struggled to identify table consumers and track how datasets were accessed across their petabyte-scale data lake. To solve this, the team built a usage attribution system using Amazon S3 server access logs to visualize time-based partition keys against access timestamps and IAM roles. These visualizations revealed distinctive patterns for daily batch jobs, backfills, and ad hoc queries, guiding migrations of thousands of tables to Apache Iceberg format. Furthermore, Yelp implemented a Default Access Retention strategy gated by IAM bucket policies to prevent accidental queries from resetting S3 Intelligent Tiering transitions. Combining these granular usage insights with targeted S3 storage classes reduced Yelp's S3 storage costs by 33 percent.

Context

In large analytics environments, data teams often lack data-driven usage attribution to identify who their stakeholders are and how data is being used. Without clear access patterns, teams rely on outdated documentation and hesitate to commit to deletion-based retention or cold storage policies that carry minimum storage durations and retrieval fees.

Approach / What changed

Yelp ingested Amazon S3 server access logs into their data lake and used batch SQL transformations to aggregate object GET operations by table, partition value, timestamp, and requester IAM role. They mapped access patterns to distinguish batch jobs, backfills, and ad hoc queries, assigned S3 Intelligent Tiering to datasets with unpredictable patterns, and implemented a Default Access Retention policy gating older partitions behind restrictive bucket IAM policies.

Takeaways

• Visualizing accessed partition dates against event timestamps reveals clear signatures: diagonal lines indicate daily batch jobs, vertical lines show backfills, and scatter points mark ad hoc queries.
• Under S3 Intelligent Tiering, objects not accessed for 30 days decrease in storage cost by 40%, and objects not accessed for 90 days decrease in storage cost by 81%.
• Default Access Retention uses restrictive IAM bucket policies on older partitions so accidental queries cannot reset S3 Intelligent Tiering tiers or trigger large retrieval fees without an approved Terraform PR.

Related reading

• Yelp · May 27, 2026

  Beyond the Menu Tree: How Yelp Built a Smarter Customer Success Chatbot with AI

  Yelp transitioned from a legacy customer support chatbot that relied on rigid phrase matching and static menus to a retrieval-augmented generation system. The updated architecture classifies incoming queries using a large language model to route users into one of five specialized workflows, including QA, Billing, Refund, Cancel, and Review. For general QA, the system uses FAISS to search an in-memory vectorstore built exclusively from article metadata such as titles, summaries, and headers rather than raw article text. Retrieved articles are supplied to the language model as prompt context, followed by automated validations for safety, character limits, and hallucinated hyperlinks. In production A/B testing, this metadata-driven design doubled the chatbot resolution rate while achieving approximately 94% recall@5.

  Lina Lee, Machine Learning Engineer; Nelson Lee, Engineering Manager

• Yelp · Mar 27, 2026