---
title: "How A/B Testing Can Help Your Team Prioritize, Not Just Validate"
description: "A/B testing can help teams prioritize projects by forcing them to estimate effect sizes and required sample sizes before committing resources. The most common mistake is calling tests too early based on statistical significance without sufficient statistical power, leading to unreliable results. To run a test correctly, teams should estimate the expected KPI improvement, use a sample size calculator to determine test duration, and decide if the wait is worthwhile. For example, a complete signup flow redesign expected to boost conversion from 10% to 15% requires 24 times less data than a color change expected to lift it to 11%, and a non-significant result can still be useful. Startups with limited traffic should focus on large, high-potential changes to avoid working towards a local maximum, while larger companies with high traffic can profitably test small optimizations."
---

# How A/B Testing Can Help Your Team Prioritize, Not Just Validate

[Intercom](https://yomu.fyi/company/intercom) · Oct 8, 2026

**Type:** Problem & solution

## Summary

A/B testing can help teams prioritize projects by forcing them to estimate effect sizes and required sample sizes before committing resources. The most common mistake is calling tests too early based on statistical significance without sufficient statistical power, leading to unreliable results. To run a test correctly, teams should estimate the expected KPI improvement, use a sample size calculator to determine test duration, and decide if the wait is worthwhile. For example, a complete signup flow redesign expected to boost conversion from 10% to 15% requires 24 times less data than a color change expected to lift it to 11%, and a non-significant result can still be useful. Startups with limited traffic should focus on large, high-potential changes to avoid working towards a local maximum, while larger companies with high traffic can profitably test small optimizations.

## Approach / What changed

Estimate the expected improvement in the key performance indicator, use a sample size calculator to determine how long the test must run, then decide whether the test is worth the time and resources based on the potential upside.

## Takeaways

- A/B tests should be used to prioritize projects by estimating effect sizes and required sample sizes before committing resources.
- Detecting small effects requires much larger sample sizes, making small optimization tests expensive for low-traffic sites.
- Startups should focus on large, high-potential changes to avoid working towards a local maximum, while larger companies can profitably test small improvements.

## FAQ

### How can I decide which A/B test to run?

Estimate the expected KPI improvement, calculate the required sample size using a calculator, and assess if the test duration is worth the potential gain.

### Why shouldn't startups run small optimization A/B tests?

Small tests require long durations to achieve statistical power, distracting from bigger projects, and the potential gain is minimal compared to larger changes.

### What should I do if an A/B test shows no significant difference?

The post says it's okay; you can still use the new version if it provides a better experience or positions you for future iteration.

**Tags:** [A/B Testing](https://yomu.fyi/topic/a-b-testing), [Experimentation](https://yomu.fyi/topic/experimentation), [Product Design](https://yomu.fyi/topic/product-design), [Startups](https://yomu.fyi/topic/startups)

- Source: [Intercom](https://www.intercom.com/blog/why-ab-tests-should-yield-more-than-results)
- Source URL: https://www.intercom.com/blog/why-ab-tests-should-yield-more-than-results
- Ingested by Yomu: 2026-10-08T21:00:35.800Z

[Read original post](https://www.intercom.com/blog/why-ab-tests-should-yield-more-than-results)