Building Jarvis Pro: Route first, answer later

Summary

Grab designed Jarvis Pro as an AI assistant for account managers turning complex account data into actionable merchant conversations. Early prototypes generated polished but operationally unsound advice, such as pushing promotions when order drops were caused by outlet pauses and fulfillment issues. To address this, the team implemented a "route first, answer later" design that uses a prompted LLM classification step constrained by a routing guide to dictate context, tool paths, answer structure, and guardrails before generation. The pipeline also isolates narrow memory decisions, reconciles metric freshness across data sources, and caps tool calls to maintain usability. In offline evaluations across 351 realistic prompts, routing matched safe routes 99.4% of the time while answer quality scores in a portfolio subset improved from 78.5 to 91.0.

Takeaways

1. 1

   In offline launch-readiness evaluations on 351 realistic prompts, routing matched the expected safe route for 99.4% of cases.

2. 2

   A 150-case portfolio and brand answer-quality evaluation subset improved its average rubric score from 78.5 to 91.0 after implementing layered routing and guardrails.

3. 3

   Data discrepancies between dashboards and warehouse tables were addressed by building regression checks that normalized source values and compared daily rows across approved metric paths.