Canva incident report: API Gateway outage

Summary

On November 12, 2024, canva.com suffered an outage lasting approximately one hour due to cascading API Gateway failures. The disruption began when a stale Cloudflare routing rule induced severe latency and packet loss between Ashburn and Singapore during an editor asset deployment. Cloudflare's concurrent streaming mechanism consolidated over 270,000 user requests for a single delayed JavaScript chunk; when the asset finally loaded, clients simultaneously generated a 1.5 million requests-per-second thundering herd to Canva's API Gateway. This surge collided with a telemetry performance regression involving lock contention on Netty event loop threads, exhausting off-heap memory and causing the Linux Out Of Memory Killer to terminate API Gateway containers. Canva mitigated the failure by temporarily blocking all traffic at the CDN level, stabilizing replacement Amazon ECS tasks, and gradually restoring traffic under strict rate limits.

Context

During a routine deployment of Canva's single-page editor application, a stale routing rule in Cloudflare caused severe network latency and 66% peak packet loss between Singapore and Ashburn for user IPv6 traffic. Because requests stalled rather than failed, canary error alarms did not trigger. Over 270,000 requests waited on a single Cloudflare cache stream for a delayed JavaScript file required to render the editor's object panel.

Approach / What changed

To recover from the resulting traffic surge and task crash loop, Canva implemented a temporary Cloudflare firewall rule blocking all incoming traffic at the CDN level and redirected users to a status page. This isolation permitted new Amazon ECS API Gateway tasks to initialize cleanly. Engineers subsequently restored traffic incrementally, starting with Australian users subjected to strict rate limits before widening access.

Takeaways

• Cloudflare's request consolidation mechanism held over 270,000 waiting requests, creating an instantaneous 1.5 million requests-per-second thundering herd upon completion.
• Lock contention introduced by metric re-registration inside an event loop degraded API Gateway throughput and exacerbated off-heap memory exhaustion.
• Canva modified its release guardrails to track page load completion events alongside JavaScript error rates during canary deployments.

Related reading

• Canva · Dec 10, 2024

  The science of routing print orders

  Canva's global print network requires selecting optimal suppliers to balance delivery times, packaging counts, and environmental emissions. To resolve these challenges before user checkout, the engineering team designed a modular routing architecture that decouples graph construction, decision logic, and path traversal. During graph traversal, the system generates action objects capturing forward paths and decision query results, which are compiled into timestamped routing logs in blob storage for asynchronous auditing. Utilizing preprocessed graph queries alongside ElastiCache, Redis, and database read replicas, the infrastructure sustains high-throughput evaluation without coupling cost logic to traversal code. As a result, print routing completes within an average of 50 milliseconds at the 99th percentile during peak usage while maintaining 99.999% data availability.

  Constantinos Kavadias

• Canva · Feb 24, 2025

  Behind the scenes of Canva's DesignDNA campaign