Performance bottlenecks of Go application on Kubernetes with non-integer (floating) CPU allocation

Summary

Grab's real-time stream processing platform encountered severe consumer lag and CPU throttling when running Go-based Kafka consumer pipelines on Kubernetes. The issue originated when the Vertical Pod Autoscaler (VPA) scaled pod CPU allocations down to floating-point values such as 1.94 cores. Because AUTO-GOMAXPROCS rounds non-integer CPU limits down to integers, Go runtime thread allocation dropped to 1 core, significantly throttling pipeline throughput despite available pod capacity. Setting a minimum floor of 2 cores instantly restored CPU utilization to 95% and cleared the message backlog. To prevent similar throttling, the team utilized integer CPU scaling recommendations available in VPA v0.13 on Kubernetes 1.25 and above.

Takeaways

1. 1

   AUTO-GOMAXPROCS rounds down non-integer CPU allocations to integer values, causing Go applications with 1.94 allocated cores to run with GOMAXPROCS set to 1 and underutilize resources.

2. 2

   When Kubernetes allocates fractional CPU below 1 core (such as 0.5 cores), AUTO-GOMAXPROCS enforces a minimum value of 1, leading to cgroup CPU starvation and throttling.

3. 3

   Vertical Pod Autoscaler v0.13 introduces support for proposing integer CPU values on Kubernetes 1.25 and above to prevent GOMAXPROCS misalignment.