Enhancing Cloud Usage Forecasting, Monitoring & Optimizing

Summary

After Etsy completed its 2020 migration from an on-premise data center to Google Cloud Platform (GCP), the resulting FinOps team developed a global approach to forecasting, monitoring, and optimizing cloud spend. Costs are grouped into core infrastructure, machine learning and product enablement, data enablement, and Dev, with trailing Cost Per Visit (CPV) serving as the primary forecasting metric after regression tests found visits most correlated with costs. BigQuery and Looker dashboards, alerts, weekly Finance comparisons, and recurring cost-owner and leadership readouts support variance detection, while costs are tracked centrally without chargebacks to preserve flexibility. Optimization work combines FinOps, engineering, and Architecture collaboration, producing GCS storage reductions through monitoring, automation, TTLs, and Intelligent tiering, plus compute, model-deployment, and network compression improvements. The next priority is better visibility into shared machine learning platform spend to improve experiment ROI analysis and identify optimization opportunities.

Takeaways

1. 1

   Visits have consistently shown the highest correlation with costs among the tested forecasting variables, but CPV does not directly capture non-human traffic, some services, or Dev, data, and machine learning training costs.

2. 2

   Cost increases are handled through source analysis, collaboration, validation, joint troubleshooting, assigned action items, and communication to Finance after confirmation or resolution.

3. 3

   A dedicated GCS optimization squad introduced usage monitoring, Data engineering automation, access-pattern-based TTLs, and Intelligent tiering; Etsy’s GCS usage is now below its level from two years earlier.