Latency Incident Report: Socket Bind Before Connect

Summary

An August 11, 2016 incident caused increased routing latency in Heroku’s EU Common Runtime, even though CPU, packet, memory, request, and socket metrics remained normal. The routing layer received EADDRINUSE while connecting to dynos, leading engineers to investigate Linux TCP port allocation, TIME_WAIT, and the Erlang VM’s system calls with strace. They found that the VM called bind() with 0.0.0.0:0 before connect(), forcing Linux to allocate an ephemeral source port without knowing the destination; this made the ephemeral port range an effective per-instance traffic limit and could fail before 65,536 connections. Testing reproduced the failures and showed that setting SO_REUSEADDR before bind(), exposed in Erlang as {reuseaddr, true}, removed the bind errors; the corresponding Vegur patch was deployed a few days later.

Takeaways

1. 1

   Calling bind() before connect() makes Linux choose an ephemeral source port without destination information, imposing a much tighter effective limit than the TCP 4-tuple model suggests.

2. 2

   Linux can return EADDRINUSE during ephemeral-port allocation before 65,536 connections because it randomly probes a limited sequence of candidate ports rather than scanning every port.

3. 3

   Setting SO_REUSEADDR before bind() resolved the reproduced failures, and the Erlang equivalent is the {reuseaddr, true} socket option.