SupabasePaul CopplestoneIncident / postmortem
Alpha Launch Postmortem
Summary
An unexpected viral submission on Hacker News prompted sudden alpha traffic for Supabase, attracting 30,000 visitors and over 1,400 signups in one week. The surge strained initial infrastructure, which initially relied on a single server running Docker Compose for middleware and Digital Ocean for customer databases. System bottlenecks emerged rapidly, including hit limits on Digital Ocean droplet counts, frequent physical server faults, and a three-hour provisioning outage when Cloudflare reached its 1,000-subdomain limit. In response, the team migrated middleware to multiple globally load-balanced ECS clusters via AWS Global Accelerator. Supabase also transitioned 1,800 servers from Digital Ocean to AWS t3a instances, nearly halving database startup times from 90 seconds to 50 seconds.
Context
During alpha testing, an unplanned viral Hacker News post drove 30,000 visitors and over 1,400 signups in seven days, pushing Supabase to over 1,000 customer databases. This unexpected traffic surge exposed infrastructure bottlenecks across third-party providers, including rapid exhaustion of Digital Ocean server limits, physical hardware errors requiring offline migrations, and a three-hour outage caused by hitting Cloudflare's 1,000-subdomain threshold.
Approach / What changed
Supabase migrated its middleware from a single Ubuntu server running Docker Compose to multiple ECS clusters load-balanced with AWS Global Accelerator. To address cloud provider issues and credit limitations on Digital Ocean, the team migrated 1,800 servers to AWS under the AWS Activate program, deploying on AWS t3a instances to improve Postgres provisioning speed.
Takeaways
Hitting Cloudflare's 1,000-subdomain limit caused a three-hour outage because account upgrade permissions were restricted solely to the account owner, preventing available team members from resolving it immediately.
Running middleware via Docker Compose on a single 4-CPU, 8GB RAM Ubuntu server unexpectedly handled the initial viral traffic before being transitioned to multiple ECS clusters using AWS Global Accelerator.
Migrating 1,800 database servers from Digital Ocean to AWS t3a instances reduced Postgres database startup times from approximately 90 seconds to 50 seconds.
Source and freshness
- Source
- supabase.com
- Published
- Added to Yomu
- Last updated
Summaries are generated from the original by a language model. Check the original before relying on it.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
The best engineering reads, once a week
One useful email. Unsubscribe anytime.