Apache Arrow Cut Snowflake Fetch Memory Growth by Up to 79%

Summary

An ML training run was exhausting worker memory before training could start because a cold Snowflake fetch peaked at 88.8 GiB RSS, versus 17.6 GiB when loading local Parquet. The shared Snowflake-to-pandas path created Python objects and SQLAlchemy Row proxies before pandas built the DataFrame, with temporary representations requiring roughly three times the final DataFrame size at peak. An Apache Arrow path for supported reads reduced median peak process RSS growth by 60–79% across three query shapes, while peak Python allocations fell by 69–87%. The narrow and wide tests also ran faster, but aggregate timing was mixed, so elapsed time was not the headline claim. Existing jobs adopted the default without query rewrites, with normalization preserving the pandas contract and SQLAlchemy retained for unsupported reads; in one workflow, lower fetch peaks doubled the training-data window on the same cluster size.

Context

An ML training run exhausted worker memory before training could start. The Snowflake fetch peaked at 88.8 GiB RSS, while loading the same data from local Parquet peaked at 17.6 GiB. The shared fetch path materialized Python objects and SQLAlchemy rows before constructing the final pandas DataFrame, creating a pervasive memory constraint across ML workflows.

Approach / What changed

An Apache Arrow fetch path was added for supported Snowflake reads and made the default through the shared data-access layer. The implementation normalized dtype and representation differences to preserve the pandas contract, retained SQLAlchemy for unsupported reads, and used compatibility checks across nine query shapes and four tables.

Takeaways

• Across three query shapes, Apache Arrow reduced median peak process RSS growth by 60–79% and peak Python allocations by 69–87%.
• Arrow eligibility required a Snowflake connection, no bound query parameters, and no unsupported pandas read_sql_query options; other reads retained the SQLAlchemy path.
• In one large workflow, the lower fetch peak doubled the training-data window on the same cluster size, moving the memory bottleneck from loading data to model training.

Related reading