# Troubleshooting Unusual AWS ELB 5XX Error

[Grab](https://yomu.fyi/company/grab) · Dharmarth Shah · May 10, 2017

**Type:** Problem & solution

## Summary

Grab experienced intermittent HTTP 5XX alerts when its Gothena service sent driver location updates to the Astrolabe service through an AWS Elastic Load Balancer (ELB). CloudWatch metrics revealed that requests were failing to reach healthy backend instances because of an uneven load distribution favoring a single ELB node in one Availability Zone. The team verified that Route 53 was properly using Alias records and ruled out OS-level DNS caching since Linux does not cache DNS queries by default. Connection inspection with netstat across multiple Go services confirmed a heavily skewed distribution of connections toward specific ELB IP addresses. Comparative tests with cURL, tcpdump, Go, Python, and Ruby in an isolated environment demonstrated that Go reused connections across requests while other runtimes opened new connections per request.

## Context

Grab's Gothena service frequently received ELB HTTP 5XX warning alerts during peak hours when sending high-volume driver location updates to the Astrolabe service.

## Approach / What changed

The team analyzed AWS CloudWatch metrics, consulted AWS support, inspected connection counts per ELB IP address using netstat across instances, and evaluated DNS and connection handling with tcpdump across cURL, Go, Python, and Ruby.

## Takeaways

- AWS ELB CloudWatch metrics showed SurgeQueue filling and SpillOver occurring in only one Availability Zone while all backend instances remained completely healthy.
- AWS Route 53 Alias Record Sets offer the functionality of CNAME records for ELBs while propagating IP changes faster and avoiding additional resolution charges.
- Linux systems do not cache DNS queries by default unless a local DNS server like dnsmasq.d is running or custom hosts file configurations are applied.

**Tags:** [AWS](https://yomu.fyi/topic/aws), [Go](https://yomu.fyi/topic/go), [Monitoring](https://yomu.fyi/topic/monitoring), [Observability](https://yomu.fyi/topic/observability), [Reliability](https://yomu.fyi/topic/reliability)

[Read original post](https://engineering.grab.com/troubleshooting-unusual-aws-elb-5xx-error)