# Canva incident report: API Gateway outage

[Canva](https://yomu.fyi/company/canva) · Brendan Humphreys · Dec 20, 2024

**Type:** Incident / postmortem

## Summary

On November 12, 2024, canva.com suffered an outage lasting approximately one hour due to cascading API Gateway failures. The disruption began when a stale Cloudflare routing rule induced severe latency and packet loss between Ashburn and Singapore during an editor asset deployment. Cloudflare's concurrent streaming mechanism consolidated over 270,000 user requests for a single delayed JavaScript chunk; when the asset finally loaded, clients simultaneously generated a 1.5 million requests-per-second thundering herd to Canva's API Gateway. This surge collided with a telemetry performance regression involving lock contention on Netty event loop threads, exhausting off-heap memory and causing the Linux Out Of Memory Killer to terminate API Gateway containers. Canva mitigated the failure by temporarily blocking all traffic at the CDN level, stabilizing replacement Amazon ECS tasks, and gradually restoring traffic under strict rate limits.

## Context

During a routine deployment of Canva's single-page editor application, a stale routing rule in Cloudflare caused severe network latency and 66% peak packet loss between Singapore and Ashburn for user IPv6 traffic. Because requests stalled rather than failed, canary error alarms did not trigger. Over 270,000 requests waited on a single Cloudflare cache stream for a delayed JavaScript file required to render the editor's object panel.

## Approach / What changed

To recover from the resulting traffic surge and task crash loop, Canva implemented a temporary Cloudflare firewall rule blocking all incoming traffic at the CDN level and redirected users to a status page. This isolation permitted new Amazon ECS API Gateway tasks to initialize cleanly. Engineers subsequently restored traffic incrementally, starting with Australian users subjected to strict rate limits before widening access.

## Takeaways

- Cloudflare's request consolidation mechanism held over 270,000 waiting requests, creating an instantaneous 1.5 million requests-per-second thundering herd upon completion.
- Lock contention introduced by metric re-registration inside an event loop degraded API Gateway throughput and exacerbated off-heap memory exhaustion.
- Canva modified its release guardrails to track page load completion events alongside JavaScript error rates during canary deployments.

**Tags:** [AWS](https://yomu.fyi/topic/aws), [Caching](https://yomu.fyi/topic/caching), [Incident Response](https://yomu.fyi/topic/incident-response), [Java](https://yomu.fyi/topic/java), [Reliability](https://yomu.fyi/topic/reliability)

[Read original post](https://www.canva.dev/blog/engineering/canva-incident-report-api-gateway-outage)