Baseline
What is a baseline?
A baseline is the score for how your AI product or workflow works today. You collect it by running your evals against a set of pre-determined inputs before you make any changes. Every experiment variant gets compared against it. If the variant gets a better score, you can be confident it's an improvement.
How do you design inputs for a baseline?
Think of your inputs like a test. If you are using golden dataset evals, the dataset already defines both the inputs and the ideal outputs. If you are using code assertions or LLM-as-a-Judge evals, you only need to define the inputs.
For a personal workflow, a small set of inputs that represent your typical usage is enough. For a production product, make sure the inputs represent what you expect from real customers. This sounds simple, but it's challenging in practice. Expect to keep evolving your input set as you learn what real customers do.
How do you collect a baseline?
There are two steps. First, run each input through your product or workflow and log the LLM responses. For a few inputs, you can do this by hand and record the outputs in a spreadsheet. For customer-facing products, automate it with a test harness: a script that runs each input through your LLM service, transforms each response into what each eval expects, runs every eval, and scores the run.
Second, run all of your evals on those outputs. The result is a baseline rate for each error category. For the Interview Coach, that looks like leading questions in 15 out of 105 suggested questions, general questions in 3 out of 105, and already-answered questions in 9 out of 105.
Related terms:
Last Updated: September 1, 2026