Cost Efficient Snowflake CI

Summary

Ramp’s data team used Snowflake CI checks that created a full duplicate production database for each pull request, causing rising warehouse costs and up to 30 minutes of developer idle time as pull-request volume and model count grew. They changed the process by storing dbt manifest.json artifacts in S3, using dbt’s state selector to build modified models, and generating an upstream-model list for cloning. A custom dbt macro uses Snowflake’s information schema to create zero-copy clones of required production tables in the pull-request database, preserving a single namespace for debugging instead of relying on defer across databases. The resulting CI command sequence builds and tests modified models plus direct downstream dependencies, and the post reports a dramatic, mostly sustained dip in the rolling seven-day average warehouse cost despite higher pull-request velocity.

Takeaways

1. 1

   Saving manifest.json after production runs enabled dbt’s --state selector to compare pull-request changes with the live production project and reduce the number of models built.

2. 2

   Snowflake zero-copy clones supplied existing upstream tables without rebuilding computationally heavy models, including lineages involving event-stream data and window functions.

3. 3

   The revised CI flow preserved a single database namespace for debugging and produced a dramatic, mostly sustained reduction in the rolling seven-day average warehouse cost despite increased pull-request velocity.