# Automatic rule backtesting with large quantities of data

[Grab](https://yomu.fyi/company/grab) · Chao Wang · Sep 8, 2022

**Type:** Problem & solution

## Summary

Evaluating new or modified risk rules previously required Grab analysts to run slow offline Presto queries, manually construct payloads, or run rules in shadow mode for days. To standardize and accelerate this workflow, Grab developed an automated backtesting system powered by an AWS EMR Spark pipeline. Historical events are continuously ingested via Kafka and a Kubernetes stream pipeline into S3 using Snappy-compressed Parquet. Users configure replay intervals and rule definitions directly in the rule engine UI, which triggers asynchronous Spark jobs through Amazon SQS and Lambda. This automated simulation replaces multi-week shadow mode runs and generates downloadable aggregation metrics on transactions, user counts, and treatment outcomes.

## Context

Testing rule performance across teams was unstandardized, inaccurate, and slow, relying on disjointed Presto queries, manual payload setups, or days of shadow-mode execution across different rule engines.

## Approach / What changed

Building a Spark-based backtesting pipeline on AWS EMR triggered from a rule engine UI via SQS and Lambda to replay rule logic against historical Kafka events stored in S3.

## Takeaways

- Replaying historical event data via Spark pipelines eliminated the need for multi-day shadow mode testing, saving weeks in the rule onboarding cycle.
- Streaming trust inference events into dedicated S3 buckets via Kubernetes pipelines improved processing efficiency and avoided data lake delays.
- Relying on data engineers to write underlying Spark analytics created a bottleneck, motivating the future development of a domain-specific language for self-service rule configuration.

**Tags:** [AWS](https://yomu.fyi/topic/aws), [Data Pipelines](https://yomu.fyi/topic/data-pipelines), [Kafka](https://yomu.fyi/topic/kafka), [Kubernetes](https://yomu.fyi/topic/kubernetes), [Streaming](https://yomu.fyi/topic/streaming)

[Read original post](https://engineering.grab.com/automatic-rule-backtesting)