Chasing that 32 ms tail latency in ClickHouse

Summary

A customer observed an extra ~32 ms tail latency on simple SELECT queries after idle periods in ClickHouse’s Native TCP protocol. The root cause was that after a TCP idle period (RTO), the congestion window resets to initcwnd (8 MSS), and with TCP_NODELAY enabled, each metadata packet (ProfileEvents, Logs, Progress) becomes a separate TCP segment, exceeding the small window and forcing an additional round trip. The ClickHouse Native protocol sends these metadata packets between Data and EndOfStream, and the unbatched writes caused the problem. The team reproduced the issue deterministically using Linux network namespaces and tc to simulate 32 ms RTT and initcwnd 8. They fixed it by batching those metadata packets into a single buffer write at the application layer, reducing tail latency from ~32 ms to ~1 ms, as confirmed by a Go client program that measured the time between the last Data packet and EndOfStream.

Takeaways

1. 1

   After a TCP idle period (RTO), the Linux congestion window resets to initcwnd, which can be as low as 8 MSS, limiting the number of unacknowledged packets per round trip.

2. 2

   With TCP_NODELAY enabled, each application write() becomes a separate TCP segment; multiple small metadata writes can exhaust the congestion window and cause an extra round trip.

3. 3

   Batching metadata packets into a single buffer write at the application layer eliminates the extra round trip and reduces tail latency from ~32 ms to ~1 ms on high-RTT connections.

FAQ