# The foundations of Canva’s continuous data platform with Snowpipe Streaming

[Canva](https://yomu.fyi/company/canva) · Jack Caperon · Jan 6, 2025

**Type:** Problem & solution

## Summary

As Canva expanded to over 200 million monthly active users, ingestion throughput reached 25 billion records daily, causing AWS Data Firehose costs to consume nearly half of the product analytics platform budget. To reduce these expenses and eliminate intermediate file staging, the engineering team integrated Snowflake's Snowpipe Streaming directly with their Java-based Kinesis Data Streams pipeline. The architecture streams records directly into Snowflake tables using logical channels with offset checkpoints while configuring client buffering up to a five-minute maximum lag. In handling Kinesis Client Library edge cases, record processors coordinate with the LeaseCoordinator to drop leases when channels become unhealthy. Operating in production for over six months, the system ingested more than 20.35 petabytes of data, decreased query latency to roughly ten minutes, and reduced overall cloud spend by 45 percent.

## Context

Canva's user base grew to over 200 million monthly active users, generating 25 billion records daily. Ingesting this data into Snowflake via AWS Data Firehose grew to account for nearly 50% of the platform's cost due to Firehose's 5 KB billing minimums and incremental charges, as well as cost increases from using Kinesis Data Streams as a source.

## Approach / What changed

Canva replaced the Firehose and S3 staging pipeline by integrating Kinesis Data Streams directly with Snowflake's Snowpipe Streaming API in Java workers. Snowpipe Streaming ingests rows directly into Snowflake tables via channels tracked with offset tokens. To optimize cost, Canva buffered data using a MAX\_CLIENT\_LAG setting of 5 minutes and handled channel invalidation failures by dropping Kinesis Client Library leases via LeaseCoordinator.

## Takeaways

- AWS Data Firehose bills in 5 KB increments with a 5 KB minimum size per record for S3 destinations, which significantly drove up costs at Canva's 25-billion-record daily volume.
- Snowpipe Streaming charges 0.01 credits per client per hour billed in 1-second increments plus intermediate file migration compute, while host-to-S3 transfers avoid extra data transfer costs.
- Because the Kinesis Client Library lacks a native restart mechanism when a Snowpipe Streaming channel fails, Canva allows record processors to declare unhealthiness and drop leases via LeaseCoordinator.

**Tags:** [Architecture](https://yomu.fyi/topic/architecture), [AWS](https://yomu.fyi/topic/aws), [Data Pipelines](https://yomu.fyi/topic/data-pipelines), [Scalability](https://yomu.fyi/topic/scalability), [Streaming](https://yomu.fyi/topic/streaming)

[Read original post](https://www.canva.dev/blog/engineering/snowpipe-streaming)