Automatic rule backtesting with large quantities of data

Summary

Evaluating new or modified risk rules previously required Grab analysts to run slow offline Presto queries, manually construct payloads, or run rules in shadow mode for days. To standardize and accelerate this workflow, Grab developed an automated backtesting system powered by an AWS EMR Spark pipeline. Historical events are continuously ingested via Kafka and a Kubernetes stream pipeline into S3 using Snappy-compressed Parquet. Users configure replay intervals and rule definitions directly in the rule engine UI, which triggers asynchronous Spark jobs through Amazon SQS and Lambda. This automated simulation replaces multi-week shadow mode runs and generates downloadable aggregation metrics on transactions, user counts, and treatment outcomes.

Context

Testing rule performance across teams was unstandardized, inaccurate, and slow, relying on disjointed Presto queries, manual payload setups, or days of shadow-mode execution across different rule engines.

Approach / What changed

Building a Spark-based backtesting pipeline on AWS EMR triggered from a rule engine UI via SQS and Lambda to replay rule logic against historical Kafka events stored in S3.

Takeaways

• Replaying historical event data via Spark pipelines eliminated the need for multi-day shadow mode testing, saving weeks in the rule onboarding cycle.
• Streaming trust inference events into dedicated S3 buckets via Kubernetes pipelines improved processing efficiency and avoided data lake delays.
• Relying on data engineers to write underlying Spark analytics created a bottleneck, motivating the future development of a domain-specific language for self-service rule configuration.

Related reading

• Grab · Jul 13, 2022

  How we automated FAQ responses at Grab

  Internal engineering on-call engineers at Grab spent substantial working hours handling repetitive questions in Slack channels, such as how-to inquiries and access permission requests. To resolve this without building an in-house tool, the team conducted an anonymized vendor comparison and selected OneBar through an employee voting process and a phased proof-of-concept. Initial rollouts were restricted by contract to 20 channels, leading the team to prioritize deployment based on Slack message volume and member counts. Populating the knowledge base required roughly a quarter of consistent updates alongside tech talks, while a targeted crowdsourcing campaign among new onboarders expanded the glossary and grew usage to approximately 3,000 users.

  Preeti Karkera

• Grab · Jun 9, 2022

  How facial recognition technology keeps you safe

  Grab utilizes facial recognition technology across its platform for driver authentication, passenger verification, and digital electronic Know Your Customer (e-KYC) processes. The core pipeline consists of image preprocessing through face detection and alignment, anti-spoofing checks, feature extraction into high-dimensional vector embeddings, and downstream verification or search. To counter spoof attacks like screen replays, synthetic moiré patterns are generated and cropped face patches are used during training and inference to focus on local structures rather than global semantic noise. Face verification challenges involving shallow ID datasets and masked faces are resolved using semi-Siamese training architectures and masked data augmentations.

  Kai Feng Tee

• Grab · May 23, 2024

  How we evaluated the business impact of marketing campaigns

  Grab needed a scalable, centralised way to evaluate the business impact of millions of marketing messages sent across expanding campaigns. Manual analysis by marketing analysts produced inconsistent results and failed to scale, leading to the creation of multi-touch attribution and controlled experiment methodologies. The team designed a Marketing attribution model (MAM) that credits touchpoints based on relevance and recency, while using stratified hold-out groups to assess single campaign lift against baseline metrics. Statistical significance calculations, implemented using the jStat library based on the central limit theorem, ensure observed business metric differences like gross merchandise value are not random. The system provides campaign managers with direct access to impact analyses while the team works on challenges surrounding smaller sample sizes and hold-out group constraints.

  Jie Zhang

• Grab · Jun 2, 2022

  Graph concepts and applications

  Real-world systems generate dynamic, non-random connections that traditional statistical approaches fail to characterize or forecast. Graph models represent these structures through vertices and edges, abstracting complex networks into mathematically tractable relationships. Common data representation formats include the Resource Description Framework (RDF), which models subject-predicate-object triples with IRIs, literals, and blank nodes, and Labeled Property Graphs (LPGs), which store arbitrary key-value properties directly on nodes and edges. Graph databases, derived from the LPG model, treat relationships with equal weight to entities, delivering responsive traversals for highly interconnected systems. While they offer agility and explicit relationship modeling, graph databases lack a standardized query language and remain poorly suited for standard transaction-focused workloads compared to relational databases.

  Wenxiang Lu