Deep Dive into Database Timeouts in Rails

Summary

Following a production outage where a database failover caused a Ruby on Rails application to exhaust its Puma server threads, an investigation was conducted to understand how ActiveRecord and MySQL timeout settings behave. A reproduction environment using Docker, Puma, and Toxiproxy replicated how hanging requests to a failing database consume all available server threads, ultimately starving unrelated endpoints. The analysis breaks down ActiveRecord connection pooling mechanics alongside underlying mysql2 and libmysqlclient settings, specifically checkout_timeout, connect_timeout, and read_timeout. Testing confirmed how existing and new TCP connections transition through socket states during network interruptions while waiting on configured timeout intervals.

Context

A database failover caused a production Rails application to slow down and freeze across all endpoints—even those not using the affected database—because worker threads became exhausted until the application servers were manually restarted.

Approach / What changed

The failure was replicated locally in a Dockerized Rails application using Puma and Toxiproxy to simulate network interruptions, tracing TCP connection states with the ss utility across varying connect_timeout and read_timeout settings.

Takeaways

• When Rails server threads are blocked waiting on database timeouts for an unavailable database, all Puma worker threads can become exhausted, starving endpoints connected to healthy databases.
• The default read_timeout from libmysqlclient is 3 × 10 minutes and the default connect_timeout is 120 seconds, allowing stalled requests to block threads for extensive periods if left unconfigured.
• If an established database connection breaks, ActiveRecord initially attempts to reuse the checked-in connection until read_timeout elapses, after which it attempts to open a new connection and waits on connect_timeout.

Related reading

• Grab · Mar 11, 2019

  Context Deadlines and How to Set Them

  Microservice architectures operating under heavy network traffic require robust timeout handling to prevent slow or failing dependencies from causing cascading failures across services. Naive, static timeout configurations across a call chain often cause downstream components to waste compute effort on requests that upstream callers have already abandoned. To establish predictable timeout thresholds, engineers can align limits with service-level latency percentiles, pairing P99 limits with median latency retry allowances. Go's context package improves upon static network timeouts by propagating request-scoped deadlines and cancellation signals across service boundaries. This distributed context ensures downstream servers recognize remaining time budgets and terminate unneeded processing immediately when parent deadlines expire or callers manually cancel requests.

  Michael Cartmell

• Grab · Oct 18, 2016

  This Rocket Ain't Stopping - Achieving Zero Downtime for Rails to Golang API Migration

  Grab transitioned its public passenger app APIs from a legacy Rails application to a Golang service-oriented architecture to consolidate its codebase and engineering teams. Initial attempts to proxy traffic through a cloned Rails server via gRPC were abandoned after encountering TCP load imbalances during autoscaling events and memory leaks in the gRPC Ruby gem. The team pivoted to direct logic migration, porting Ruby logic directly into Go while decomposing modules into standalone services. Verification relied on log-based load testing and live shadow testing, where write operations were safely validated using mock data access layers that evaluated expected database outcomes. Production rollout progressed endpoint-by-endpoint using requests-per-second traffic throttling and prewarmed AWS Elastic Load Balancers before executing the final DNS switch.

  Lian Yuanlin

• Grab · May 10, 2017

  Troubleshooting Unusual AWS ELB 5XX Error

  Grab experienced intermittent HTTP 5XX alerts when its Gothena service sent driver location updates to the Astrolabe service through an AWS Elastic Load Balancer (ELB). CloudWatch metrics revealed that requests were failing to reach healthy backend instances because of an uneven load distribution favoring a single ELB node in one Availability Zone. The team verified that Route 53 was properly using Alias records and ruled out OS-level DNS caching since Linux does not cache DNS queries by default. Connection inspection with netstat across multiple Go services confirmed a heavily skewed distribution of connections toward specific ELB IP addresses. Comparative tests with cURL, tcpdump, Go, Python, and Ruby in an isolated environment demonstrated that Go reused connections across requests while other runtimes opened new connections per request.

  Dharmarth Shah

• Grab · May 5, 2019

  Preventing Pipeline Calls from Crashing Redis Clusters

  A single Redis slave node failure caused Grab's Apollo booking service to suffer an over 95 percent call failure rate for one minute despite running a three-shard cluster with two replicas per partition. Investigation revealed that the service configured Go-Redis to route all read queries exclusively to slave nodes to offload master CPU usage. When the slave node dropped offline, batched HMGET pipeline calls failed completely because the client wrapper treated a single command failure as a failure of the entire pipeline. Furthermore, the Go-Redis client cached cluster topology and only lazily refreshed state every sixty seconds, continuing to direct traffic to the dead replica until the timer expired. Grab addressed this risk by recommending dedicated pipeline clients configured with latency-based routing to allow reads to fall back to responsive master nodes.

  Michael Cartmell