DNS Resolution in Go and Cgo

Summary

Go applications experiencing load balancing issues across AWS Elastic Load Balancer (ELB) nodes trace uneven traffic distribution to IP address sorting defined in RFC 6724. Comparing Go's native DNS resolver with Cgo and glibc's getaddrinfo shows that both initially sort destination addresses using Rule 9 longest matching prefix rules. Disabling IPv6 on the network interface causes C and Cgo resolvers to return IP addresses in randomized order, while the native Go resolver continues deterministic sorting. Examination of net/addrselect.go reveals that Go's native resolver implements only a subset of the RFC rules and omits dynamic source address selection. Achieving permanent parity requires modifying the Go source code directly.

Context

A load balancing issue occurred on AWS Elastic Load Balancer (ELB) nodes due to deterministic IP address sorting under RFC 6724.

Approach / What changed

Investigated DNS resolution across native Go, Cgo, and C by comparing getaddrinfo behavior with Go's net package, disabling interface-level IPv6, and reviewing net/addrselect.go.

Takeaways

• Go's native DNS resolver implements only a subset of RFC 6724 rules and lacks dynamic source address selection.
• Disabling IPv6 on the network interface randomizes address order for Cgo and C resolvers using getaddrinfo, but does not affect Go's native resolver.
• Cgo executes C code on an operating system thread rather than on a lightweight goroutine.

Related reading

• Grab · May 10, 2017

  Troubleshooting Unusual AWS ELB 5XX Error

  Grab experienced intermittent HTTP 5XX alerts when its Gothena service sent driver location updates to the Astrolabe service through an AWS Elastic Load Balancer (ELB). CloudWatch metrics revealed that requests were failing to reach healthy backend instances because of an uneven load distribution favoring a single ELB node in one Availability Zone. The team verified that Route 53 was properly using Alias records and ruled out OS-level DNS caching since Linux does not cache DNS queries by default. Connection inspection with netstat across multiple Go services confirmed a heavily skewed distribution of connections toward specific ELB IP addresses. Comparative tests with cURL, tcpdump, Go, Python, and Ruby in an isolated environment demonstrated that Go reused connections across requests while other runtimes opened new connections per request.

  Dharmarth Shah

• Grab · Oct 18, 2016

  This Rocket Ain't Stopping - Achieving Zero Downtime for Rails to Golang API Migration

  Grab transitioned its public passenger app APIs from a legacy Rails application to a Golang service-oriented architecture to consolidate its codebase and engineering teams. Initial attempts to proxy traffic through a cloned Rails server via gRPC were abandoned after encountering TCP load imbalances during autoscaling events and memory leaks in the gRPC Ruby gem. The team pivoted to direct logic migration, porting Ruby logic directly into Go while decomposing modules into standalone services. Verification relied on log-based load testing and live shadow testing, where write operations were safely validated using mock data access layers that evaluated expected database outcomes. Production rollout progressed endpoint-by-endpoint using requests-per-second traffic throttling and prewarmed AWS Elastic Load Balancers before executing the final DNS switch.

  Lian Yuanlin

• Grab · Sep 27, 2016

  Round-robin in Distributed Systems

  Building client-side load balancing for Grab's Common Data Service prompted a move from AWS Elastic Load Balancers to DNS discovery due to persistent connection issues and unpredictable scaling events. After patching an open-source library that failed to rotate IP sequences properly, the author evaluated different Go patterns for round-robin routing. A mutex-protected array counter provides the simplest model for basic retrieval, though adding mutations requires careful lock coordination. Alternatively, a dedicated balancer goroutine receiving requests over nested channels enables explicit operation timeouts and centralized event handling at the cost of higher code complexity and channel creation overhead. The author recommends the mutex approach for resource fetching and the goroutine-based design for workload balancing.

  Gao Chao

• Grab · Mar 5, 2019

  Structured Logging: The Best Friend You’ll Want When Things Go Wrong

  Grab redesigned its backend logging approach to address mounting vendor costs, query language limitations, and debugging difficulties across a growing microservices ecosystem. Most services previously emitted syslog-style key-value logs almost entirely at the INFO level, which made volume reduction difficult and lacked causal ordering and automated correlation. The engineering team migrated to a self-managed Elastic stack backend and built a structured logging library in Go from the ground up. This framework introduces dynamic log-level adjustment at runtime, automatic trace-based log correlation via Grab-Kit, and the Common Grab Log Schema to enforce consistent JSON formatting without Elasticsearch indexing conflicts.

  Aditya Praharaj