Test Harness
What is a test harness?
A test harness is a script that automates a full eval run. It calls your LLM service for each input, logs the output, transforms each response into the input each eval expects, runs each eval against the response, and scores the whole run. Teresa Torres built hers as a python script for her Interview Coach. The output is an error rate for each error category.
Why does a test harness matter for customer-facing AI products?
Evals only help you improve your product if you can rerun the same inputs after every change. Every time you change a prompt, a model, or an orchestration pattern, you run your inputs through your product, score the outputs with your evals, and compare the variant to your baseline. Doing that by hand works once, not experiment after experiment. For customer-facing products, Teresa strongly recommends automating this step.
How does a test harness run each eval?
Each eval is a module that tells the harness how to execute it: how to turn the raw LLM output into the input that eval expects, and whether running it means custom code (a code assertion) or an LLM call (an LLM-as-a-Judge). Each experiment variant is documented in a yaml config file with an ID, what changed, a commit sha, and which evals to run.
When can a personal workflow skip a test harness?
For personal workflows with a few inputs, run them by hand and log the LLM responses in a spreadsheet, or ask your favorite model to write a python script that runs each input and saves the output to a text file. A test harness earns its keep when the same inputs get run over and over.
Related terms:
Last Updated: September 1, 2026