Grab Bench: Evaluating AI on Grab-shaped production work

Summary

Grab Bench is a configurable evaluation harness for assessing AI systems on production-shaped work where plausible outputs can violate metric definitions, tool contracts, evidence boundaries, or hidden code invariants. It uses YAML configuration, task plugins, provider and modality checks, row-level records, model summaries, and deterministic scorers or LLM judges according to each task. Synthetic or redacted cases preserve production-relevant difficulty while protecting live data; profile reasoning requires ontology-valid JSON claims and faithful evidence IDs, while coding tasks exercise hidden tests and anti-gaming checks. The harness also records tokens, latency, skip reasons, configurations, and failure tags so teams can diagnose whether errors reflect unsupported claims, parameter drift, hidden invariants, cost, or capability gaps. It concludes that synthetic evaluations test contract adherence under controlled pressure but do not establish production uplift.

Takeaways

1. 1

   Passenger-profile cases require strict JSON, ontology-valid claims, existing evidence IDs, grounded support, calibrated confidence, and safe actions; fabricated IDs, cite-all behavior, or unsupported sensitive claims receive penalties or score caps.

2. 2

   Agentic coding evaluations test more than visible correctness: they include hidden workspace tests and checks for API compatibility, stateful invariants, idempotency, concurrency, cancellation, oversized patches, and other shortcut patterns.

3. 3

   The benchmark treats synthetic evaluation as evidence of contract adherence under controlled pressure, not proof of production uplift; live retrieval quality, user impact, and rollout decisions require separate evidence.