Troubleshooting Unusual AWS ELB 5XX Error

Summary

Grab experienced intermittent HTTP 5XX alerts when its Gothena service sent driver location updates to the Astrolabe service through an AWS Elastic Load Balancer (ELB). CloudWatch metrics revealed that requests were failing to reach healthy backend instances because of an uneven load distribution favoring a single ELB node in one Availability Zone. The team verified that Route 53 was properly using Alias records and ruled out OS-level DNS caching since Linux does not cache DNS queries by default. Connection inspection with netstat across multiple Go services confirmed a heavily skewed distribution of connections toward specific ELB IP addresses. Comparative tests with cURL, tcpdump, Go, Python, and Ruby in an isolated environment demonstrated that Go reused connections across requests while other runtimes opened new connections per request.

Context

Grab's Gothena service frequently received ELB HTTP 5XX warning alerts during peak hours when sending high-volume driver location updates to the Astrolabe service.

Approach / What changed

The team analyzed AWS CloudWatch metrics, consulted AWS support, inspected connection counts per ELB IP address using netstat across instances, and evaluated DNS and connection handling with tcpdump across cURL, Go, Python, and Ruby.

Takeaways

• AWS ELB CloudWatch metrics showed SurgeQueue filling and SpillOver occurring in only one Availability Zone while all backend instances remained completely healthy.
• AWS Route 53 Alias Record Sets offer the functionality of CNAME records for ELBs while propagating IP changes faster and avoiding additional resolution charges.
• Linux systems do not cache DNS queries by default unless a local DNS server like dnsmasq.d is running or custom hosts file configurations are applied.

Related reading

• Grab · May 24, 2017

  DNS Resolution in Go and Cgo

  Go applications experiencing load balancing issues across AWS Elastic Load Balancer (ELB) nodes trace uneven traffic distribution to IP address sorting defined in RFC 6724. Comparing Go's native DNS resolver with Cgo and glibc's getaddrinfo shows that both initially sort destination addresses using Rule 9 longest matching prefix rules. Disabling IPv6 on the network interface causes C and Cgo resolvers to return IP addresses in randomized order, while the native Go resolver continues deterministic sorting. Examination of net/addrselect.go reveals that Go's native resolver implements only a subset of the RFC rules and omits dynamic source address selection. Achieving permanent parity requires modifying the Go source code directly.

  Ryan Law

• Grab · Dec 28, 2015

  The Curious Case of the Phantom Instance

  Datadog dashboards for the grab_attention cluster displayed periodic 1.5X step increases in Elastic Load Balancer (ELB) health check requests and ElastiCache Redis connections, creating the illusion of an untracked instance running outside Auto Scaling Group records. Inspecting instance hostname tags revealed that the existing two instances were simply receiving elevated ping counts from the load balancers. AWS Support clarified that ELB scaling events provision new nodes while keeping old nodes running for roughly 90 minutes to handle cached DNS clients. Investigation also revealed that two separate ELBs were attached to the cluster, altering expected request baselines. Furthermore, the application's health check endpoint initiated a non-pooled Redis connection on every request, directly translating load balancer pings into database connection spikes.

  Lian Yuanlin

• Grab · Jan 7, 2018

  Dealing with the Meltdown Patch at Grab

  AWS infrastructure maintenance related to Meltdown patches led to severe CPU utilization spikes across Grab's ElastiCache Redis instances. Because Redis is single-threaded, spikes past 50% CPU on two-vCPU instances threatened service capacity, and initial Multi-AZ failovers only provided temporary relief until the new master nodes received rolling patches. To handle the increased overhead before their peak traffic window, the engineering team horizontally scaled both clustered and non-clustered Redis fleets. For Redis 3.2.4 clusters lacking live re-sharding support, they provisioned larger clusters, warmed caches, and redirected traffic. Non-clustered workloads were resolved by provisioning extra nodes, migrating compatible services to Redis Cluster, or updating application code to shard data across multiple instances.

  Althaf Hameez

• Grab · Sep 27, 2016

  Round-robin in Distributed Systems

  Building client-side load balancing for Grab's Common Data Service prompted a move from AWS Elastic Load Balancers to DNS discovery due to persistent connection issues and unpredictable scaling events. After patching an open-source library that failed to rotate IP sequences properly, the author evaluated different Go patterns for round-robin routing. A mutex-protected array counter provides the simplest model for basic retrieval, though adding mutations requires careful lock coordination. Alternatively, a dedicated balancer goroutine receiving requests over nested channels enables explicit operation timeouts and centralized event handling at the cost of higher code complexity and channel creation overhead. The author recommends the mutex approach for resource fetching and the goroutine-based design for workload balancing.

  Gao Chao