Make Better Decisions by Embracing Uncertainty

Summary

Ramp uses multi-armed bandits to choose among third-party bank-linking providers when provider performance varies by financial institution and available data is noisy, incomplete, or stale. Instead of maintaining separate, static A/B tests, it applies Thompson Sampling, modeling each provider’s success probability with a Beta distribution and repeatedly sampling, routing traffic to the highest sampled value, then updating the model from the result. Simulations examined changing success-rate trends, noise, volume, rolling-window size, trend type, and initial conditions; the algorithm generally exploited clearly better providers, split traffic more evenly when rates were close, and recovered eventually from incorrect starting conditions. In a sequential quasi-experiment, Thompson Sampling increased financial-institution linking success by 10% and reduced customers with manually uploaded bank statements by 25%.

Context

Ramp needed to select among third-party bank-linking providers for each financial institution, despite noisy and imperfect information, changing performance, and uncertainty caused by limited or stale attempt data. Static A/B testing would require separate experiments for thousands of institutions and would still require decisions about how to route traffic after each experiment.

Approach / What changed

Ramp used Thompson Sampling, a multi-armed bandit method, to balance exploration and exploitation. It modeled each provider's success probability with a Beta distribution, sampled a value for each provider at every decision point, routed traffic to the highest sample, and updated the selected provider's distribution after observing the result. The team evaluated behavior through simulations and a sequential quasi-experiment.

Takeaways

• Thompson Sampling uses wider Beta distributions for uncertain providers, allowing lower-performing options with greater uncertainty to receive exploratory traffic.
• Simulations found that the algorithm quickly favored clearly better providers, split traffic more evenly when success rates were close, and eventually recovered from incorrect initial conditions.
• The rollout produced a 10% increase in financial-institution linking success and a 25% decrease in customers with manually uploaded bank statements.

Related reading