Automatic rule backtesting with large quantities of data

Summary

Evaluating new or modified risk rules previously required Grab analysts to run slow offline Presto queries, manually construct payloads, or run rules in shadow mode for days. To standardize and accelerate this workflow, Grab developed an automated backtesting system powered by an AWS EMR Spark pipeline. Historical events are continuously ingested via Kafka and a Kubernetes stream pipeline into S3 using Snappy-compressed Parquet. Users configure replay intervals and rule definitions directly in the rule engine UI, which triggers asynchronous Spark jobs through Amazon SQS and Lambda. This automated simulation replaces multi-week shadow mode runs and generates downloadable aggregation metrics on transactions, user counts, and treatment outcomes.

Takeaways

1. 1

   Replaying historical event data via Spark pipelines eliminated the need for multi-day shadow mode testing, saving weeks in the rule onboarding cycle.

2. 2

   Streaming trust inference events into dedicated S3 buckets via Kubernetes pipelines improved processing efficiency and avoided data lake delays.

3. 3

   Relying on data engineers to write underlying Spark analytics created a bottleneck, motivating the future development of a domain-specific language for self-service rule configuration.