Grab Bench: Evaluating AI on Grab-shaped production work

Summary

AI models evaluated on internal tasks often produce subtly plausible errors, such as altering metric definitions in SQL queries or missing hidden stateful invariants in code patches. To address this, Grab built Grab Bench, a configurable evaluation harness that executes model providers through specialized task plugins. The platform evaluates synthetic or redacted datasets that preserve production constraints across domains like query generation, tool use, passenger-profile reasoning, and agentic coding. By recording granular row-level outputs with deterministic scorers and LLM judges, the harness exposes failure modes rather than relying solely on aggregate leaderboards.

Takeaways

• Grab Bench utilizes synthetic datasets paired with deterministic scorers to test strict domain contracts—such as ontology checks, evidence IDs, and hidden workspace tests—without exposing production data.
• The evaluation harness tests for shortcuts by running weak baselines like cite-all-evidence and visible-test-only agents to ensure the benchmark cannot be trivially passed.
• Evaluation artifacts are divided into open teaching artifacts for understanding the harness and restricted certification artifacts with hidden splits to prevent prompt overfitting.

Related reading

• Grab · May 29, 2026

  From decentralized Docs-as-Code to a centralized repository: Evolving Grab's documentation strategy

  Grab originally adopted a decentralized Docs-as-Code workflow to manage Markdown documentation alongside service code, but the model strained as engineering scaled. Dispersed repositories led to fragmented terminology, uncoordinated repository moves, uneven linting standards, and noisy search results. To resolve these issues, the organization transitioned to a centralized documentation repository with standardized templates, automated CI quality gates, and a unified search overlay combining fuzzy title search with Glean. Consolidating Markdown into a single repository also established a clean, structured knowledge base tailored for internal AI agents. The centralized setup fostered continuous maintenance, averaging 27 daily commits and an update merged roughly every 50 minutes.

  Karen Kue

• Grab · Jun 21, 2026

  Palana (Part 2): Architecting isolation, identity, and auditability for AI agents

  Grab's Palana platform provisions isolated, Kubernetes-native runtime environments for autonomous AI agents using dedicated per-agent namespaces and role-based access controls. The architecture separates network enforcement across layers, applying Layer 3 and Layer 4 containment with Cilium and NetworkPolicy alongside Layer 7 application filtering evaluated by Open Policy Agent. Agent interactions with large language models route through a LiteLLM proxy wrapper that retrieves credentials from HashiCorp Vault based on Kubernetes pod context rather than client headers. Secrets management is divided between directly readable agent paths and proxy-only placeholder paths that prevent raw tokens from residing in runtime filesystems. An automated reaper monitors multi-source activity signals to shut down idle compute resources while preserving persistent storage and configuration state.

  Kevin Littlejohn

• Grab · Jul 3, 2026

  Migrating Counter Service storage: Design choices and learnings

  Grab migrated its Counter Service storage backend from a legacy wide-column database to Aerospike to support real-time anti-fraud windowed aggregations across tens of thousands of queries per second. To decouple storage from the Rust reader service, engineers introduced a storage facade using enum dispatch, avoiding the per-query heap allocations of boxed trait objects. The reader used configuration-driven operating modes to support shadow reads and deterministic traffic splitting without requiring code updates. On the write path, the schema was redesigned to collapse multiple bucket records into a single sorted map per counter, using atomic server-side operations to increment counters and prune expired entries. This data model redesign reduced total record counts and in-memory primary index usage by over an order of magnitude while enabling a zero-downtime transition.

  Jia Long Loh

• Grab · Aug 21, 2026

  Building Jarvis Pro: Route first, answer later

  Grab designed Jarvis Pro as an AI assistant for account managers turning complex account data into actionable merchant conversations. Early prototypes generated polished but operationally unsound advice, such as pushing promotions when order drops were caused by outlet pauses and fulfillment issues. To address this, the team implemented a "route first, answer later" design that uses a prompted LLM classification step constrained by a routing guide to dictate context, tool paths, answer structure, and guardrails before generation. The pipeline also isolates narrow memory decisions, reconciles metric freshness across data sources, and caps tool calls to maintain usability. In offline evaluations across 351 realistic prompts, routing matched safe routes 99.4% of the time while answer quality scores in a portfolio subset improved from 78.5 to 91.0.

  Christian Coffrant