# Performance bottlenecks of Go application on Kubernetes with non-integer (floating) CPU allocation

[Grab](https://yomu.fyi/company/grab) · Shubham Badkur · May 23, 2023

**Type:** Problem & solution

## Summary

Grab's real-time stream processing platform encountered severe consumer lag and CPU throttling when running Go-based Kafka consumer pipelines on Kubernetes. The issue originated when the Vertical Pod Autoscaler (VPA) scaled pod CPU allocations down to floating-point values such as 1.94 cores. Because AUTO-GOMAXPROCS rounds non-integer CPU limits down to integers, Go runtime thread allocation dropped to 1 core, significantly throttling pipeline throughput despite available pod capacity. Setting a minimum floor of 2 cores instantly restored CPU utilization to 95% and cleared the message backlog. To prevent similar throttling, the team utilized integer CPU scaling recommendations available in VPA v0.13 on Kubernetes 1.25 and above.

## Context

Stream processing pipelines running Go applications on Kubernetes encountered CPU throttling and consumer lag when Vertical Pod Autoscaler (VPA) allocated non-integer CPU cores.

## Approach / What changed

Adjusting the minimum VPA limit to integer values (such as 2 cores) for resource-heavy pipelines and adopting VPA v0.13 (Kubernetes ≥1.25) to propose integer-only CPU scaling recommendations.

## Takeaways

- AUTO-GOMAXPROCS rounds down non-integer CPU allocations to integer values, causing Go applications with 1.94 allocated cores to run with GOMAXPROCS set to 1 and underutilize resources.
- When Kubernetes allocates fractional CPU below 1 core (such as 0.5 cores), AUTO-GOMAXPROCS enforces a minimum value of 1, leading to cgroup CPU starvation and throttling.
- Vertical Pod Autoscaler v0.13 introduces support for proposing integer CPU values on Kubernetes 1.25 and above to prevent GOMAXPROCS misalignment.

**Tags:** [Go](https://yomu.fyi/topic/go), [Kafka](https://yomu.fyi/topic/kafka), [Kubernetes](https://yomu.fyi/topic/kubernetes), [Performance](https://yomu.fyi/topic/performance), [Streaming](https://yomu.fyi/topic/streaming)

[Read original post](https://engineering.grab.com/performance-bottlenecks-go-apps)