Building resilient real-time streaming workers with Amazon DynamoDB leases

Summary

Managing hundreds of persistent outbound WebSocket connections requires explicit ownership because a failed worker can drop connections and stop data flow until they are reassigned. The described fleet-management pattern uses Amazon DynamoDB leases, conditional writes, and a global secondary index to ensure one worker owns each connection, renew ownership with heartbeats, and find expired leases for reconciliation. Amazon ECS on AWS Fargate workers poll Amazon SQS for notifications, maintain the WebSockets, and release or transfer ownership during graceful shutdown, failures, and scaling. The design also records desired state, the upstream URL, and the last processed sequence number in the lease item, while clock synchronization supports its time-based expiry. The post identifies heartbeat writes as the dominant DynamoDB cost and recommends provisioned capacity with Auto Scaling for sustained high connection counts, while noting that longer intervals reduce cost but extend failover.

Takeaways

1. 1

   DynamoDB conditional writes provide distributed coordination without a separate lock service; a condition on update_item confirms that one worker still owns a connection before changing its lease.

2. 2

   A global secondary index on desired_state and lease_expires_at_ms lets workers query for STARTED connections with expired leases and reconcile connections left unmanaged by failed workers.

3. 3

   Heartbeat writes dominate DynamoDB cost. Increasing the heartbeat interval lowers WCU consumption but requires a proportionally longer lease, increasing the failover window.