Migrating Counter Service storage: Design choices and learnings

Summary

Grab migrated its Counter Service storage backend from a legacy wide-column database to Aerospike to support real-time anti-fraud windowed aggregations across tens of thousands of queries per second. To decouple storage from the Rust reader service, engineers introduced a storage facade using enum dispatch, avoiding the per-query heap allocations of boxed trait objects. The reader used configuration-driven operating modes to support shadow reads and deterministic traffic splitting without requiring code updates. On the write path, the schema was redesigned to collapse multiple bucket records into a single sorted map per counter, using atomic server-side operations to increment counters and prune expired entries. This data model redesign reduced total record counts and in-memory primary index usage by over an order of magnitude while enabling a zero-downtime transition.

Takeaways

• To avoid per-query heap allocations associated with boxed trait objects at high query volumes, the Rust reader service implemented its storage abstraction using a concrete facade with enum dispatch.
• Collapsing time-series buckets into a single KEY_ORDERED_MAP record per counter resolved memory bottlenecks caused by Aerospike's 64-byte in-memory primary index per record.
• Instead of relying on record-level expiry that would drop an entire counter's history, the write path explicitly pruned stale bucket entries using atomic MapRemoveByKeyRangeOp calls during increments.

Related reading

• Grab · Aug 21, 2026

  Building Jarvis Pro: Route first, answer later

  Grab designed Jarvis Pro as an AI assistant for account managers turning complex account data into actionable merchant conversations. Early prototypes generated polished but operationally unsound advice, such as pushing promotions when order drops were caused by outlet pauses and fulfillment issues. To address this, the team implemented a "route first, answer later" design that uses a prompted LLM classification step constrained by a routing guide to dictate context, tool paths, answer structure, and guardrails before generation. The pipeline also isolates narrow memory decisions, reconciles metric freshness across data sources, and caps tool calls to maintain usability. In offline evaluations across 351 realistic prompts, routing matched safe routes 99.4% of the time while answer quality scores in a portfolio subset improved from 78.5 to 91.0.

  Christian Coffrant

• Grab · Aug 12, 2026

  Grab Bench: Evaluating AI on Grab-shaped production work

  AI models evaluated on internal tasks often produce subtly plausible errors, such as altering metric definitions in SQL queries or missing hidden stateful invariants in code patches. To address this, Grab built Grab Bench, a configurable evaluation harness that executes model providers through specialized task plugins. The platform evaluates synthetic or redacted datasets that preserve production constraints across domains like query generation, tool use, passenger-profile reasoning, and agentic coding. By recording granular row-level outputs with deterministic scorers and LLM judges, the harness exposes failure modes rather than relying solely on aggregate leaderboards.

  Christian Coffrant

• Grab · Jun 21, 2026

  Palana (Part 2): Architecting isolation, identity, and auditability for AI agents

  Grab's Palana platform provisions isolated, Kubernetes-native runtime environments for autonomous AI agents using dedicated per-agent namespaces and role-based access controls. The architecture separates network enforcement across layers, applying Layer 3 and Layer 4 containment with Cilium and NetworkPolicy alongside Layer 7 application filtering evaluated by Open Policy Agent. Agent interactions with large language models route through a LiteLLM proxy wrapper that retrieves credentials from HashiCorp Vault based on Kubernetes pod context rather than client headers. Secrets management is divided between directly readable agent paths and proxy-only placeholder paths that prevent raw tokens from residing in runtime filesystems. An automated reaper monitors multi-source activity signals to shut down idle compute resources while preserving persistent storage and configuration state.

  Kevin Littlejohn

• Grab · May 29, 2026

  From decentralized Docs-as-Code to a centralized repository: Evolving Grab's documentation strategy

  Grab originally adopted a decentralized Docs-as-Code workflow to manage Markdown documentation alongside service code, but the model strained as engineering scaled. Dispersed repositories led to fragmented terminology, uncoordinated repository moves, uneven linting standards, and noisy search results. To resolve these issues, the organization transitioned to a centralized documentation repository with standardized templates, automated CI quality gates, and a unified search overlay combining fuzzy title search with Glean. Consolidating Markdown into a single repository also established a clean, structured knowledge base tailored for internal AI agents. The centralized setup fostered continuous maintenance, averaging 27 daily commits and an update merged roughly every 50 minutes.

  Karen Kue