Troubleshooting Unusual AWS ELB 5XX Error

Summary

Grab experienced intermittent HTTP 5XX alerts when its Gothena service sent driver location updates to the Astrolabe service through an AWS Elastic Load Balancer (ELB). CloudWatch metrics revealed that requests were failing to reach healthy backend instances because of an uneven load distribution favoring a single ELB node in one Availability Zone. The team verified that Route 53 was properly using Alias records and ruled out OS-level DNS caching since Linux does not cache DNS queries by default. Connection inspection with netstat across multiple Go services confirmed a heavily skewed distribution of connections toward specific ELB IP addresses. Comparative tests with cURL, tcpdump, Go, Python, and Ruby in an isolated environment demonstrated that Go reused connections across requests while other runtimes opened new connections per request.

Takeaways

1. 1

   AWS ELB CloudWatch metrics showed SurgeQueue filling and SpillOver occurring in only one Availability Zone while all backend instances remained completely healthy.

2. 2

   AWS Route 53 Alias Record Sets offer the functionality of CNAME records for ELBs while propagating IP changes faster and avoiding additional resolution charges.

3. 3

   Linux systems do not cache DNS queries by default unless a local DNS server like dnsmasq.d is running or custom hosts file configurations are applied.