---
title: "How to Use Quasi-experiments and Counterfactuals to Build Great Products"
description: "Correlation does not establish causation, so the post presents causal inference as a way to determine why product metrics change and to validate or invalidate business strategies. The proposed evidence ladder places randomized A/B tests first, quasi-experiments such as fixed-effects regression and difference-in-differences second, and counterfactual estimation third, with descriptive statistics providing no direct causal evidence. It explains that quasi-experiments use non-random treatment and control divisions, while difference-in-differences depends on a parallel-trends assumption and fixed-effects regression assumes relevant dividing factors were collected. For a security update released to everyone, the team trained a time-series model on unaffected feature usage and global activity trends, then found no lift versus actual usage, indicating no negative effect on usage. Because observational methods bring greater uncertainty and frequent false positives, the post recommends stating assumptions, relaxing them through robustness checks, and using DAGs and Dagitty to test causal hypotheses."
---

# How to Use Quasi-experiments and Counterfactuals to Build Great Products

[Shopify](https://yomu.fyi/company/shopify) · 2023-10-18 · Sep 28, 2020

**Type:** Explainer

## Summary

Correlation does not establish causation, so the post presents causal inference as a way to determine why product metrics change and to validate or invalidate business strategies. The proposed evidence ladder places randomized A/B tests first, quasi-experiments such as fixed-effects regression and difference-in-differences second, and counterfactual estimation third, with descriptive statistics providing no direct causal evidence. It explains that quasi-experiments use non-random treatment and control divisions, while difference-in-differences depends on a parallel-trends assumption and fixed-effects regression assumes relevant dividing factors were collected. For a security update released to everyone, the team trained a time-series model on unaffected feature usage and global activity trends, then found no lift versus actual usage, indicating no negative effect on usage. Because observational methods bring greater uncertainty and frequent false positives, the post recommends stating assumptions, relaxing them through robustness checks, and using DAGs and Dagitty to test causal hypotheses.

## Context

Product teams need to distinguish causal relationships from correlations when evaluating changes and business strategies. A/B tests are not always possible because of tooling limitations, implementation time, ethical concerns, or past launches without experimentation. When randomization is unavailable, quasi-experiments or counterfactual methods provide alternatives, though with greater uncertainty.

## Approach / What changed

The post organizes causal inference methods into an evidence ladder, then explains randomized A/B tests, quasi-experiments using fixed-effects regression and difference-in-differences, and counterfactual estimation with time-series models. It recommends robustness checks that state and relax assumptions, including DAG-based analysis with Dagitty. A security-update example uses unaffected feature usage and global activity trends to model a counterfactual control group.

## Takeaways

- Difference-in-differences requires a control group whose pre-treatment trend is parallel to the treatment group; the post-treatment break is then attributed to treatment under that assumption.
- A time-series counterfactual model estimated the security update’s expected usage without a control group and found no lift between predicted and actual usage, suggesting the update did not reduce usage.
- Quasi-experiments and counterfactuals produce more uncertainty and frequent false positives than randomized experiments, making robustness checks essential.

**Tags:** [Data Analytics](https://yomu.fyi/topic/data-analytics), [Machine Learning](https://yomu.fyi/topic/machine-learning), [Testing](https://yomu.fyi/topic/testing)

- Source: [Shopify](https://shopify.engineering/using-quasi-experiments-counterfactuals)
- Source URL: https://shopify.engineering/using-quasi-experiments-counterfactuals
- Ingested by Yomu: 2026-08-31T01:11:35.015Z

[Read original post](https://shopify.engineering/using-quasi-experiments-counterfactuals)