Experiments

Q: How do experiments work for AI products and workflows?

Because an LLM can give different answers to the same input, you measure a change across many inputs rather than one. Collect a baseline by running a fixed set of inputs through your product and scoring the outputs with your evals. Then design an experiment variant (a documented set of changes to the prompt, context, model, or orchestration, grouped around one hypothesis), run the same inputs through it, score with the same evals, and compare. Keep what lowers the error rate without worsening another category, roll back what doesn't, and expect several experiments before a real improvement.

What are experiments?

Experiments are structured activities designed to support or refute a hypothesis by testing whether a specific change has the expected impact. In product discovery, the term "experiments" typically refers to large-scale randomized controlled experiments (like A/B tests) that measure the impact of something after it's been built. In AI product work, an experiment compares a changed product or workflow against a baseline using evals.

What's the difference between experiments and assumption tests?

Teams rarely have time to run real experiments in discovery. They can run A/B tests after they build something, but only if they have enough traffic. And this is not the best discovery activity.

Teams don't want to do all the work to build something before they learn they built the wrong thing. This is why assumption testing makes more sense during discovery—assumption tests are faster and take less work.

Assumption tests are used in discovery when trying to decide between ideas. Experiments are used to measure the impact of what teams built.

When should teams use experiments?

While experimentation is becoming more pervasive in product development, teams need to be smart about when and how they use experiments. Teams should use lighter-weight assumption tests during discovery and save full experiments for measuring the impact of what they deliver.

How do experiments work for AI products and workflows?

With an LLM in the loop, one output can't tell you whether a change helped, because the same input can produce different answers, so you measure across many inputs. First, collect a baseline: run a fixed set of inputs through your product and score the outputs with your evals. Then design an experiment variant: documented changes to the prompt, context, model, or orchestration, grouped around one hypothesis. Run the same inputs through the variant, score with the same evals, and compare. Keep what lowers errors without worsening another category; roll back what doesn't. Expect several experiments before a real improvement.

Learn more:

← Back to Product Discovery Glossary

Last Updated: September 2, 2026