A/B Testing
What is A/B testing?
A/B testing (also called split testing) is a quantitative research method where you test two or more variations of a design or feature with a large sample of users to see which performs better. The test randomly assigns users to different groups, with each group experiencing a different variation, and measures which version achieves better results on key metrics like conversion rates.
A/B tests are great for uncovering how a large population behaves and which design converts better. However, an A/B test will tell you which variation performs better but won't tell you why.
When should teams use A/B testing?
A/B testing is valuable, but it's actually a measurement method—not a discovery method. The problem with using A/B testing for discovery is that you have to build the thing first before you can test it. Teams often over-rely on A/B testing when there are faster ways to learn through prototyping and other discovery methods.
Teams should use lighter-weight assumption tests during discovery to decide between ideas, and save A/B tests for measuring the impact of what they've delivered. Running A/B tests properly requires checking for statistical significance, knowing when you can and can't trust the results, and having enough traffic to get meaningful data.
What are the limitations of A/B testing?
While A/B testing is quantitative and goes broad with large samples, it can be challenging to uncover the why behind user actions. An A/B test tells you which design converts better but doesn't explain the reasoning behind user behavior.
Additionally, many teams have a strong bias toward building to learn via A/B testing when there are much faster ways to learn earlier in the discovery process—before investing time in building complete features.
Learn more:
- Assumption Testing: Everything You Need to Know to Get Started
- Hypothesis Testing
- How Much Time Should You Spend in Product Discovery?
Related terms:
- Experiments
- Assumption Testing
- Hypothesis
Last Updated: October 25, 2025