Performance bottlenecks of Go application on Kubernetes with non-integer (floating) CPU allocation

Summary

Grab's real-time stream processing platform encountered severe consumer lag and CPU throttling when running Go-based Kafka consumer pipelines on Kubernetes. The issue originated when the Vertical Pod Autoscaler (VPA) scaled pod CPU allocations down to floating-point values such as 1.94 cores. Because AUTO-GOMAXPROCS rounds non-integer CPU limits down to integers, Go runtime thread allocation dropped to 1 core, significantly throttling pipeline throughput despite available pod capacity. Setting a minimum floor of 2 cores instantly restored CPU utilization to 95% and cleared the message backlog. To prevent similar throttling, the team utilized integer CPU scaling recommendations available in VPA v0.13 on Kubernetes 1.25 and above.

Context

Stream processing pipelines running Go applications on Kubernetes encountered CPU throttling and consumer lag when Vertical Pod Autoscaler (VPA) allocated non-integer CPU cores.

Approach / What changed

Adjusting the minimum VPA limit to integer values (such as 2 cores) for resource-heavy pipelines and adopting VPA v0.13 (Kubernetes ≥1.25) to propose integer-only CPU scaling recommendations.

Takeaways

• AUTO-GOMAXPROCS rounds down non-integer CPU allocations to integer values, causing Go applications with 1.94 allocated cores to run with GOMAXPROCS set to 1 and underutilize resources.
• When Kubernetes allocates fractional CPU below 1 core (such as 0.5 cores), AUTO-GOMAXPROCS enforces a minimum value of 1, leading to cgroup CPU starvation and throttling.
• Vertical Pod Autoscaler v0.13 introduces support for proposing integer CPU values on Kubernetes 1.25 and above to prevent GOMAXPROCS misalignment.

Related reading

• Grab · Feb 20, 2024

  The journey of building a comprehensive attribution platform

  Grab needed to modernize its marketing analytics from manual ad hoc queries and high data latency to a platform supporting real-time attribution for pricing models like cost per order. The engineering team initially deployed a pure stream-processing engine using Kappa architecture, Kafka, ScyllaDB, and Redis, which reduced latency from days to minutes and merged ads and promo touchpoints. However, stream-only processing faced high costs, out-of-order event issues, and difficulties running multi-touch models across longer historical windows. Grab transitioned to a Lambda architecture pairing Coban stream processing with Spark-based batch ETL and Amazon S3. This hybrid design separated real-time operational metrics from historical batch reporting, cutting real-time processing costs by approximately 25% while maintaining under 1% data discrepancy.

  Kang Huang

• Grab · Feb 1, 2024

  Managing dynamic marketplace content at scale: Grab's approach to content moderation

  Grab manages content moderation across GrabFood and GrabMart, where merchants update over 100,000 item listings daily across eight countries. The platform must adhere to internal rules, local government regulations, and external platform guidelines across these markets. To handle scale and regional nuance, Grab uses an in-house automated system powered by algorithms and machine learning to scan daily listings for violations. Items flagged as requiring subjective judgment or cultural awareness are routed to human moderators for manual review. Additionally, Grab collaborates with Google to maintain compliance with Play Store policies.

  Poonam Gambhire

• Grab · Sep 23, 2024

  Bringing Grab’s Live Activity to Android: Enhancing user experience through custom notifications

  Grab designed an equivalent to iOS Live Activities for Android to provide real-time order tracking outside the app. Because Android lacks Apple's native ActivityKit push token system, the team substituted push tokens with placeholder values to maintain technical consistency across platforms and preserve backend push targeting via their Hedwig service. For the user interface, engineering selected custom notifications over floating views because custom notifications avoid intrusive screen usage and do not require the 'Draw over other apps' permission. The client implementation separates responsibilities across dedicated classes: LiveActivityIntegrationManager handles token registration across business verticals, LiveActivityAttributes encapsulates UI configuration, and LiveActivityManager maps payloads to Android NotificationManager instances. The solution launched for Food, Mart, Express, and Transport verticals.

  Jessica Sean

• Grab · Sep 22, 2023

  Stepping up marketing for advertisers: Scalable lookalike audience

  Grab's legacy lookalike audience generation platform suffered from long creation SLAs of two working days, high costs, and low weekly update frequencies. To resolve these bottlenecks, the engineering team designed an embedding-based platform powered by an in-memory retrieval service and automated update pipelines. The system creates audience representations by averaging constituent passenger embeddings, determining user membership through real-time cosine score thresholds. To eliminate feature store latency while fitting all embeddings into memory, a hash-based compression method cuts passenger embedding storage needs by roughly 90%. Consequently, audience availability dropped to within 15 minutes of campaign creation, audience generation costs fell by 98%, and ad impressions and clicks doubled.

  William Wu