LLM-as-a-Judge
What is an LLM-as-a-Judge eval?
An LLM-as-a-Judge eval is where you use a second LLM to evaluate the output of your first LLM. You send the output to the judge along with evaluation criteria and ask for a verdict. It is one of four common types of evals, alongside golden datasets, code assertions, and customer feedback.
When should you use an LLM-as-a-Judge eval?
Only when counting the error truly requires judgment. Semantic qualities like whether a question is leading can't be checked with a string search. LLM-as-Judges let you measure open-ended output at scale without exhaustive human review.
But they are slow and cost money, even with small models. Try a code assertion first. Or use one as a filter and send only the failures to the judge.
How do you make an LLM-as-a-Judge eval work well?
Four rules. Give the judge a much simpler task than the original model: one narrow question at a time. Define your criteria so the judge returns a binary answer—true/false, yes/no—not a score. Align the judge with your own judgment before you trust it. And account for the judge's own error when you report your eval's error rate.
The fact checker for the Lovable interview stories got one claim and one transcript: is the claim grounded? The Interview Coach's leading question judge gets one question: does it assume something about the participant or signal a preferred response?
How do you validate an LLM-as-a-Judge eval?
If you can't trust the first LLM, why trust the second? Because you check it against a human. Grade some traces yourself, run the judge on the same traces, and compare its verdicts to yours, counting true positives, false positives, and false negatives. Until the judge agrees with you, don't trust its verdicts.
Learn more:
- Building My First AI Product: 6 Lessons from My 90-Day Deep Dive
- How I Designed & Implemented Evals for Product Talk's Interview Coach
Related terms:
Last Updated: September 1, 2026