Loading…
Topic
Snowflake
7 posts about Snowflake. Every summary links to the original.
- 7 posts
- 5 companies
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
Newest first
Posts
How we built a personal data scientist for every Notion employee
Notion built Data Scout, an internal data agent connecting Snowflake and Notion, letting employees query governed data in plain language and cutting request turnaround from days to minutes. Within two months, weekly data team requests fell by about 80 percent, and roughly half of all employees now use it. The agent enforces Snowflake permissions per user and uses Notion business context, enabling teams such as Sales, Engineering, and Customer Experience to ask questions about account adoption, onboarding drop-off, or support volume using shared definitions. Using MCP, teams built Data Scout Reporter for scheduled reports and domain-specific agents that reuse the same context. Notion learned correct SQL could still answer the wrong question due to four failure modes, so they built governed access, curated context, instructions, query tracing, and a review process where change evidence is not approval.
Snowflake and Anthropic announce $200M partnership
Anthropic and Snowflake announced a multi-year, $200 million partnership that brings Claude models to Snowflake’s more than 12,600 customers across Amazon Bedrock, Google Cloud Vertex AI, and Microsoft Azure. The agreement also includes a joint go-to-market initiative for deploying AI agents in large enterprises. Claude can analyze structured and unstructured data through Snowflake Cortex AI, while Snowflake Intelligence and Cortex Agents support natural-language questions, multimodal SQL-based analysis, and custom multi-agent solutions under Snowflake Horizon Catalog governance. Snowflake’s internal benchmarks report greater than 90% accuracy on complex text-to-SQL tasks, and examples include Simon Data and Intercom. The partnership is positioned to help regulated industries move AI agents from pilots to production within governed data environments.
ClickHouse Cloud vs. Snowflake: What drives the real-time performance-per-dollar gap
The benchmark compares ClickHouse Cloud with Snowflake under a continuous real-time workload, asking whether ingested data becomes query-ready quickly enough for interactive analytics. Both systems received the same 113.2-billion-row NBBO dataset at a target of 1 million rows per second, with matched sorting or clustering keys, pre-aggregations, query schedules, and approximately 16 CPUs of query-serving compute. ClickHouse used native asynchronous inserts and maintained ordering and pre-aggregation during ingestion, while Snowflake used Snowpipe Streaming and Interactive MV refresh as separately metered services. In the end-to-end CostBench comparison, ClickHouse Cloud delivered 412× better performance per dollar: Snowflake’s path cost 3.32× more and its matched queries took 124× longer in total. The comparison attributes the gap to query-readiness work, refresh lag, reconciliation, and growing drill-down latency.
How we think about text classification in the LLM era
Medium revisited its aging NSFW text-classification model, which had outdated code, infrequent retraining, and poorly documented training and performance history. The replacement uses an LLM prompt containing NSFW guidelines and story content, requesting a 0-to-100 score, structured output, and a short reason rather than requiring a training set. To evaluate it, the team created a curator-labeled dataset with deliberate oversampling of NSFW stories, then compared several LLMs with the legacy model using false-positive and false-negative metrics. Some models matched or exceeded the legacy model; the team selected Mixtral 8×7b based on cost and performance and ran inference in Snowflake. The post presents this as a strong fit where training data is costly, trends are relatively stable, and only distribution-eligible stories need scoring, while topic and spam models still benefit from continuously updated datasets.
Apache Arrow Cut Snowflake Fetch Memory Growth by Up to 79%
An ML training run was exhausting worker memory before training could start because a cold Snowflake fetch peaked at 88.8 GiB RSS, versus 17.6 GiB when loading local Parquet. The shared Snowflake-to-pandas path created Python objects and SQLAlchemy Row proxies before pandas built the DataFrame, with temporary representations requiring roughly three times the final DataFrame size at peak. An Apache Arrow path for supported reads reduced median peak process RSS growth by 60–79% across three query shapes, while peak Python allocations fell by 69–87%. The narrow and wide tests also ran faster, but aggregate timing was mixed, so elapsed time was not the headline claim. Existing jobs adopted the default without query rewrites, with normalization preserving the pandas contract and SQLAlchemy retained for unsupported reads; in one workflow, lower fetch peaks doubled the training-data window on the same cluster size.
Enabling Multi-Region Data Systems at Notion
Notion’s data residency work keeps customer data processed and stored in the region where it was generated, supporting EU requirements across Search, analytics, scheduled events, and AI. To make this possible, Notion modularized infrastructure around isolated private networks and workspace IDs, then deployed regional data lakes, Kafka clusters, Spark jobs, Elasticsearch, and vector databases. Debezium captures Postgres changes into regional Kafka, while Airflow remains a centralized orchestration layer that routes jobs to regional EMR clusters; event logging similarly routes traffic by workspace and sanitizes data before Snowflake. Centralized mapping tables, shared region helpers, Terraform, and localized sanitization let teams retain a unified Snowflake environment without exporting customer data. The work took months, and Notion says it is applying these patterns to Japan and South Korea while aiming to provision future regional data stacks within days rather than months.
Cost Efficient Snowflake CI
Ramp’s data team used Snowflake CI checks that created a full duplicate production database for each pull request, causing rising warehouse costs and up to 30 minutes of developer idle time as pull-request volume and model count grew. They changed the process by storing dbt manifest.json artifacts in S3, using dbt’s state selector to build modified models, and generating an upstream-model list for cloning. A custom dbt macro uses Snowflake’s information schema to create zero-copy clones of required production tables in the pull-request database, preserving a single namespace for debugging instead of relying on defer across databases. The resulting CI command sequence builds and tests modified models plus direct downstream dependencies, and the post reports a dramatic, mostly sustained dip in the rolling seven-day average warehouse cost despite higher pull-request velocity.
Related topics
All topics →Weekly digest
The best engineering reads, once a week.