Using Propensity Score Matching to Uncover Shopify Capital’s Effect on Business Growth

Summary

Shopify’s data team sought to estimate whether accepting Shopify Capital for the first time increased merchants’ subsequent sales, without randomly denying funding in an A/B test. They compared US first-time adopters in January 2019 with Canadian shops that later pre-qualified for Capital, using propensity scores built from Shopify sales and performance characteristics and matching them with a caliper-based greedy nearest-neighbor algorithm. After matching about 600 treated shops against roughly 8,000 controls, they evaluated log-transformed cumulative GMV from February through July 2019 with a binary regression and checked balance using standardized mean differences, visual comparisons, and variance ratios. The estimated geometric-average GMV was 36% higher for US merchants, with a 95% confidence interval of 13% to 65%; robustness checks supported a positive treatment effect.

Takeaways

1. 1

   The treatment group contained about 600 US shops adopting Capital for the first time in January 2019, while the control group contained approximately 8,000 Canadian shops that later pre-qualified for Capital.

2. 2

   Caliper matching restricted nearest-neighbor matches to a defined maximum propensity-score distance, allowing unmatched shops rather than accepting distant comparisons; all but one treated shop found a Canadian counterpart.

3. 3

   US merchants had a 36% higher geometric average of cumulative six-month GMV after first adopting Capital, with a 95% confidence interval ranging from 13% to 65%.