False Positive
What is a false positive?
A false positive is when a test or judge flags a problem that does not exist. The check says something is wrong, but when you inspect the actual output, there is no error. The term applies to any kind of evaluation—a medical test, a spam filter, a code assertion—but it becomes especially important when you use one LLM to judge the output of another.
In AI evals, a false positive means the judge reported an error that was not there. For example, a judge asked to find missed sub-groupings in a generated opportunity solution tree might report that samples are missing a sub-grouping even when there is no missing sub-group.
Why do false positives make a judge untrustworthy?
False positives inflate your error rate. If the judge counts errors that do not exist, you can't tell how often the real error actually happens, and every measurement built on that judge becomes suspect.
False positives often have their own root cause. In one calibration effort, a judge kept flagging non-existent errors because it was getting confused by unrelated upstream errors—problems introduced in earlier LLM calls that had nothing to do with the error the judge was asked to find. When those upstream errors were removed, the judge correctly ruled there was no error.
How do you catch false positives?
Specificity is the measure that catches false positives. Specificity measures the judge's accuracy when there is no error. Run the judge against a calibration set of manually labeled samples and check how often it correctly says "no error" on the clean ones. A judge with low specificity is producing false positives, and you can't trust its error counts until you fix it.
Related terms:
Learn more:
Last Updated: September 16, 2026