Experiment Variant
What is an experiment variant?
An experiment variant is a documented set of changes to your AI product or workflow that you run against the same inputs and score with the same evals, so you can compare the results to your baseline. Once your evals have measured how often each error category shows up, a variant is how you try to reduce those error rates.
A variant can include more than one change, but group your changes around a theory or hypothesis rather than trying a bunch of different things at once.
What kinds of changes go into an experiment variant?
Start with one error and ask how you might reduce it:
- Change the prompt.
- Add context.
- Change the model.
- Break a complex LLM call into a series of simpler calls.
- Make an orchestration change, like adding agentic tool calling or audit loops.
How do you document and evaluate an experiment variant?
Before you change anything, decide how you'll document the changes, because you'll want to roll back when an experiment fails. For customer-facing products, document each variant in a yaml config file with an experiment ID, a description of what changed, the commit sha that captures the code and prompts, and which evals to run.
Then run the experiment: run your inputs through the new variant, run your evals on the outputs, and compare the results to your baseline. Look at changes across all of your evals. A prompt change might improve the error rate in one category but make another one worse. The goal is to get better across the board.
If the first variant doesn't succeed, don't be discouraged. It often takes multiple experiments to find an improvement.
Related terms:
Last Updated: September 1, 2026