Cost Efficient Snowflake CI

Summary

Ramp’s data team used Snowflake CI checks that created a full duplicate production database for each pull request, causing rising warehouse costs and up to 30 minutes of developer idle time as pull-request volume and model count grew. They changed the process by storing dbt manifest.json artifacts in S3, using dbt’s state selector to build modified models, and generating an upstream-model list for cloning. A custom dbt macro uses Snowflake’s information schema to create zero-copy clones of required production tables in the pull-request database, preserving a single namespace for debugging instead of relying on defer across databases. The resulting CI command sequence builds and tests modified models plus direct downstream dependencies, and the post reports a dramatic, mostly sustained dip in the rolling seven-day average warehouse cost despite higher pull-request velocity.

Context

Ramp’s data team grew to nearly 20 additional members and maintained thousands of data models while pull-request volume increased. Its initial CI process created a complete production duplicate for every pull request, leading to unnecessary model builds, rising Snowflake warehouse costs, and developer wait time of up to 30 minutes.

Approach / What changed

The team stored production dbt manifest.json files in S3, used dbt’s --state selector to identify modified models, listed their upstream models, and seeded that list into CI. A custom dbt macro queried Snowflake’s production information schema and created zero-copy clones of the required upstream tables in the pull-request database. CI then built and tested modified models and direct downstream dependencies in one namespace.

Takeaways

• Saving manifest.json after production runs enabled dbt’s --state selector to compare pull-request changes with the live production project and reduce the number of models built.
• Snowflake zero-copy clones supplied existing upstream tables without rebuilding computationally heavy models, including lineages involving event-stream data and window functions.
• The revised CI flow preserved a single database namespace for debugging and produced a dramatic, mostly sustained reduction in the rolling seven-day average warehouse cost despite increased pull-request velocity.

Related reading