# Deep Dive into Database Timeouts in Rails

[Grab](https://yomu.fyi/company/grab) · Jia Hao Goh · Jan 29, 2018

**Type:** Explainer

## Summary

Following a production outage where a database failover caused a Ruby on Rails application to exhaust its Puma server threads, an investigation was conducted to understand how ActiveRecord and MySQL timeout settings behave. A reproduction environment using Docker, Puma, and Toxiproxy replicated how hanging requests to a failing database consume all available server threads, ultimately starving unrelated endpoints. The analysis breaks down ActiveRecord connection pooling mechanics alongside underlying mysql2 and libmysqlclient settings, specifically checkout\_timeout, connect\_timeout, and read\_timeout. Testing confirmed how existing and new TCP connections transition through socket states during network interruptions while waiting on configured timeout intervals.

## Context

A database failover caused a production Rails application to slow down and freeze across all endpoints—even those not using the affected database—because worker threads became exhausted until the application servers were manually restarted.

## Approach / What changed

The failure was replicated locally in a Dockerized Rails application using Puma and Toxiproxy to simulate network interruptions, tracing TCP connection states with the ss utility across varying connect\_timeout and read\_timeout settings.

## Takeaways

- When Rails server threads are blocked waiting on database timeouts for an unavailable database, all Puma worker threads can become exhausted, starving endpoints connected to healthy databases.
- The default read\_timeout from libmysqlclient is 3 × 10 minutes and the default connect\_timeout is 120 seconds, allowing stalled requests to block threads for extensive periods if left unconfigured.
- If an established database connection breaks, ActiveRecord initially attempts to reuse the checked-in connection until read\_timeout elapses, after which it attempts to open a new connection and waits on connect\_timeout.

**Tags:** [Docker](https://yomu.fyi/topic/docker), [Incident Response](https://yomu.fyi/topic/incident-response), [MySQL](https://yomu.fyi/topic/mysql), [Performance](https://yomu.fyi/topic/performance), [Reliability](https://yomu.fyi/topic/reliability)

[Read original post](https://engineering.grab.com/deep-dive-into-database-timeouts-in-rails)