LLM-as-a-Judge
What is an LLM-as-a-Judge eval?
An LLM-as-a-Judge eval is where you use a second LLM to evaluate the output of your first LLM. You send the output to the judge along with evaluation criteria and ask for a verdict. It is one of four common types of evals, alongside golden datasets, code assertions, and customer feedback.
When should you use an LLM-as-a-Judge eval?
Only when counting the error truly requires judgment. Semantic qualities like whether a question is leading can't be checked with a string search. LLM-as-Judges let you measure open-ended output at scale without exhaustive human review.
But they are slow and cost money, even with small models. Try a code assertion first. Or use one as a filter and send only the failures to the judge.
How do you make an LLM-as-a-Judge eval work well?
Four rules. Give the judge a much simpler task than the original model: one narrow question at a time. Define your criteria so the judge returns a binary answer—true/false, yes/no—not a score. Calibrate the judge against your own judgment before you trust it. And account for the judge's own error when you report your eval's error rate.
The fact checker for the Lovable interview stories got one claim and one transcript: is the claim grounded? The Interview Coach's leading question judge gets one question: does it assume something about the participant or signal a preferred response?
How do you calibrate an LLM-as-a-Judge eval?
If you can't trust the first LLM, why trust the second? Because you calibrate it against your own judgment. Build a calibration set—samples you label yourself, some with the error and some without—then run the judge on it. Specificity measures the judge's accuracy when there is no error; recall measures its accuracy when there is one. Until the judge agrees with you, don't trust its verdicts.
Even then, calibration isn't static. A calibrated judge can stop holding up against production data—resample its calibration set from real production traces. Judges also get confused by upstream errors in their input: one judge flagged missed groupings that weren't there because a child opportunity simply restated its parent. Fix or filter those errors before you trust the count.
Learn more:
- 4 New Evals and 16 Experiment Variants to Fix One Customer Complaint
- AI Evals: A Hands-On Guide for Product Teams
- Building My First AI Product: 6 Lessons from My 90-Day Deep Dive
- How I Designed & Implemented Evals for Product Talk's Interview Coach
Related terms:
- AI Evals
- Human Grader
- Code Assertion
- Customer Feedback Eval
- Semantic Task
- Hallucination
- Calibration Set
- Specificity
- Recall
- False Positive
- False Negative
Last Updated: September 16, 2026