Specificity
What is specificity?
Specificity measures how well a model or classifier performs on negative cases—the cases where the thing it is looking for is absent. In machine learning terms, it's the proportion of true negatives the model correctly identifies. A model with high specificity rarely raises false alarms. A model with low specificity produces false positives—it flags errors that are not there.
How does specificity apply to LLM-as-a-Judge evals?
When you calibrate an LLM-as-a-Judge against a set of samples you've manually labeled, specificity measures the judge's accuracy when there is no error. When a sample is clean, how often does the judge correctly say so? It's one of two measures to check before trusting a judge—the other is recall, the judge's accuracy when there is an error.
Why does low specificity make a judge untrustworthy?
A judge with low specificity inflates your error counts with false positives. For example, one judge built to catch missed sub-groupings in AI-generated opportunity solution trees scored 100% on recall but only 43.75% on specificity. In the cases where there was no error, the judge thought there was no error just 43.75% of the time. It correctly identified every real error, but it also identified errors where there were none.
You can't trust that judge's scoring, because most of what it flags may not be real. Any error rate it reports will be badly inflated. Before you rely on a judge to measure an error, it has to prove it stays quiet when the output is actually correct.
Related terms:
Learn more:
Last Updated: September 16, 2026