Scaling Radar: Fighting fraud with agent-assisted machine learning

Summary

Stripe Radar’s Shield model detects transaction fraud using network-wide signals. To improve recall, Stripe expanded training from roughly 800 million to 11 billion transactions, requiring optimizations that cut each epoch from 24 to 4.8 hours through streaming data loading, torch.compile, lower-precision arithmetic, concurrent execution of ResNeXt branches, and memory reductions. The larger dataset initially improved performance, but delayed fraud reports caused label immaturity, forcing a rollback because the model learned late-reported fraud as legitimate. Stripe built Agentic MLE, enabling AI agents to autonomously investigate failures, implement features, train models, and evaluate results using model explanations and prior experiment data. Using these agents, Stripe identified a training configuration change combining stochastic weight averaging at epoch 0 and shortening training to eight epochs, improving Shield’s recall by over 8% and estimating tens of millions in additional fraud prevention annually.

Takeaways

1. 1

   Expanding training data from 800 million to 11 billion transactions required infrastructure optimizations including streaming data loading, torch.compile, lower-precision arithmetic, concurrent execution of ResNeXt branches, and memory reductions from 1.3 TB to 500 GB, cutting epoch time from 24 to 4.8 hours.

2. 2

   Label immaturity from delayed fraud reports caused the larger model to initially appear better but later perform worse, forcing a rollback because it learned late-reported fraud as legitimate.

3. 3

   Agentic MLE enabled AI agents to autonomously run experiments, leading to a training configuration change (stochastic weight averaging at epoch 0 and shorter training) that improved Shield’s recall by over 8%.

FAQ