Highly concurrent in-memory counter in GoLang

Summary

High database CPU utilization caused by relentless write traffic for marketing campaign counts prompted Grab to build an in-memory aggregation buffer rather than migrate from SQL to NoSQL. The team implemented an in-memory counter in Go that batches usage updates and periodically flushes them to persistent storage. To eliminate the serialization bottlenecks of mutex-locked maps under high concurrency, the design adopted Go's sync.Map using atomic CompareAndSwap retries for value updates alongside LoadAndDelete for periodic background flushes. Because a finite set of campaign keys is repeatedly accessed, operations hit the internal read map fast path nearly 99% of the time, achieving a threefold throughput improvement over standard mutex locks in benchmarks. In production, this architecture decreased database update queries by 68% and reduced master database CPU utilization from 35% to 18%.

Takeaways

• Using sync.Map with atomic CompareAndSwap retry loops enables lock-free concurrent updates on existing keys, delivering roughly three times higher throughput than sync.RWMutex in benchmarks with 2,000 keys.
• Flushing in-memory counters via sync.Map's LoadAndDelete extracts and removes key values atomically without locking the entire map or blocking concurrent increments.
• Buffering non-critical approximate counters in application memory cut production database update traffic from 140 QPS to 45 QPS and lowered master CPU utilization by 48.5%.

Related reading

• Grab · Feb 27, 2025

  TechDocs at Grab: Cultivating a culture of quality documentation

  Engineering organizations frequently struggle with fragmented documentation, stale content, and a lack of clear ownership across disparate tools. To address these issues, Grab established TechDocs on its central Helix platform, embedding a Docs-as-Code workflow into daily engineering routines. Feedback gathered from quantitative surveys and one-on-one sessions shaped governance policies, separating stable platform documentation stored in GitLab from collaborative artifacts like RFCs in Confluence. To sustain document freshness, the platform assigns mandatory points of contact, displays last-updated timestamps, and flags pages untouched for more than three months.

  David Khu

• Grab · Jan 16, 2026

  From deployment slop to production reality: How BriX bridges the gap with enterprise-grade AI infrastructure

  Internal AI prototypes frequently fail enterprise rollouts due to diverging versions, security oversights, hardcoded credentials, and infrastructure bottlenecks. BriX addresses this deployment gap by turning AI rollout into a configuration-driven platform rather than an engineering rewrite. Built on a synchronous streaming architecture, it routes user prompts through a React frontend using Server-Sent Events, a FastAPI gateway, and LangGraph orchestration. The platform integrates model switching, centralized prompt locks, and standardized Model Context Protocols for governed enterprise data access.

  Sneh Agrawal

• Grab · Apr 11, 2025

  Evaluating performance impact of removing Redis-cache from a Scylla-backed service

  Grab operates a high-throughput Rust read service that aggregates counter metrics from Scylla tables across minutely, hourly, and daily granularities. The service initially cached aggregated responses in Redis using keys rounded to 15-minute intervals alongside a five-minute TTL. Because incoming queries predominantly requested recent data, transitions between 15-minute windows caused simultaneous cache misses across active configurations, resulting in severe Scylla traffic spikes, latency surges, and timeouts. To resolve the load imbalance, engineers proposed removing the Redis cache entirely and relying directly on Scylla's native internal caching. The rollout was staged in production by deterministically disabling Redis caching for specific counter configurations using mathematical operations on configuration IDs.

  Md Riyadh

• Grab · Apr 25, 2025

  Introducing the SOP-driven LLM agent frameworks

  Standard operating procedure (SOP) driven Large Language Model agent frameworks address common generative AI challenges such as hallucinations, non-standard output formats, and branching navigation errors. Workflows are represented as hierarchical trees where nodes encapsulate actions or decision points that can be created using a visual editor and annotated with explicit external function calls. Execution relies on a tripartite architecture consisting of a Depth-First Search planner module with backtracking, an adaptive worker agent that limits API exposure and compresses context, and a multilingual user agent. Supporting tools include a Graph Retrieval-Augmented Generation pipeline, a plugin system integrating Python and SQL, and a state stack for pausing workflows during human intervention. In production deployments for fraud and account takeover investigations, the framework automated up to 87% of cases while cutting handling times substantially.

  Fujiao Liu