A/B Test
What is an A/B test?
A/B tests are large-scale randomized controlled experiments that test whether code changes had the intended impact by comparing two variations and measuring the difference in performance. They're an effective way to validate whether a feature produced the results you expected.
A/B testing has become popular for good reason—it's a powerful measurement tool. When you can run an A/B test, you get clear data about what actually happened when real users encountered your changes. But A/B tests come with significant constraints that make them better suited for validation than discovery.
When should teams use A/B tests?
A/B tests work well for gating releases and confirming impact after you've built something. If you want to know whether a feature had the intended effect, an A/B test can tell you definitively. But you need sufficient traffic to yield statistically significant results, which many teams don't have.
The bigger limitation is timing: A/B tests require building almost everything before learning whether it was the right thing to build. All the learning happens after you've already invested in development. This makes them expensive as discovery tools—they answer "did this work?" but can't help you avoid building the wrong thing in the first place.
How do A/B tests differ from assumption tests?
A/B testing is often the default "experiment" teams think of, but it's rarely the best discovery activity. Discovery requires learning quickly and cheaply before committing to building. Assumption tests use prototyping techniques—smoke screens, Wizard of Oz tests, concierge tests—to simulate experiences and test ideas without building production code.
While A/B tests compare two built variations with real users, assumption tests let you compare multiple concepts with small samples before writing code. This shifts learning earlier in the process, where you still have time to change direction without sunk costs pulling you toward a predetermined solution.
Learn more:
- Why You Aren’t Learning As Much As You Could From Your Experiments
- Assumption Testing: Everything You Need to Know to Get Started
Related terms:
- Assumption Test
- Assumption Testing
- Experiments
- Prototype
Last Updated: October 25, 2025