Deep Dive into Database Timeouts in Rails

Summary

Following a production outage where a database failover caused a Ruby on Rails application to exhaust its Puma server threads, an investigation was conducted to understand how ActiveRecord and MySQL timeout settings behave. A reproduction environment using Docker, Puma, and Toxiproxy replicated how hanging requests to a failing database consume all available server threads, ultimately starving unrelated endpoints. The analysis breaks down ActiveRecord connection pooling mechanics alongside underlying mysql2 and libmysqlclient settings, specifically checkout_timeout, connect_timeout, and read_timeout. Testing confirmed how existing and new TCP connections transition through socket states during network interruptions while waiting on configured timeout intervals.

Takeaways

1. 1

   When Rails server threads are blocked waiting on database timeouts for an unavailable database, all Puma worker threads can become exhausted, starving endpoints connected to healthy databases.

2. 2

   The default read_timeout from libmysqlclient is 3 × 10 minutes and the default connect_timeout is 120 seconds, allowing stalled requests to block threads for extensive periods if left unconfigured.

3. 3

   If an established database connection breaks, ActiveRecord initially attempts to reuse the checked-in connection until read_timeout elapses, after which it attempts to open a new connection and waits on connect_timeout.