Using Propensity Score Matching to Uncover Shopify Capital’s Effect on Business Growth

Summary

Shopify’s data team sought to estimate whether accepting Shopify Capital for the first time increased merchants’ subsequent sales, without randomly denying funding in an A/B test. They compared US first-time adopters in January 2019 with Canadian shops that later pre-qualified for Capital, using propensity scores built from Shopify sales and performance characteristics and matching them with a caliper-based greedy nearest-neighbor algorithm. After matching about 600 treated shops against roughly 8,000 controls, they evaluated log-transformed cumulative GMV from February through July 2019 with a binary regression and checked balance using standardized mean differences, visual comparisons, and variance ratios. The estimated geometric-average GMV was 36% higher for US merchants, with a 95% confidence interval of 13% to 65%; robustness checks supported a positive treatment effect.

Context

The team wanted to determine whether Shopify Capital had a measurable effect on merchants’ future cumulative gross merchandise value. A randomized A/B test would have required automatically rejecting some interested merchants, which the team considered unacceptable. Canadian merchants served as an alternate comparison because Capital was not available there in 2019.

Approach / What changed

The study used propensity score matching to compare US first-time Capital adopters in January 2019 with Canadian shops that later pre-qualified for Capital. Features based on Shopify sales and performance were used to estimate propensity scores, with many covariates logarithmically transformed. Caliper-based greedy nearest-neighbor matching produced balanced groups, followed by binary regression on log-transformed six-month GMV and robustness checks including an A/A simulation and bootstrapping.

Takeaways

• The treatment group contained about 600 US shops adopting Capital for the first time in January 2019, while the control group contained approximately 8,000 Canadian shops that later pre-qualified for Capital.
• Caliper matching restricted nearest-neighbor matches to a defined maximum propensity-score distance, allowing unmatched shops rather than accepting distant comparisons; all but one treated shop found a Canadian counterpart.
• US merchants had a 36% higher geometric average of cumulative six-month GMV after first adopting Capital, with a 95% confidence interval ranging from 13% to 65%.

Related reading