Code Assertion

What is a code assertion?

A code assertion (also called a code-based assertion) is a type of AI eval where you use traditional deterministic code to evaluate the quality of an LLM response. Some teams call them code-based evals. Like unit tests, they are virtually free and the code works as intended every time.

Code assertions are one of four common types of evals, alongside golden datasets, LLM-as-a-Judge evals, and customer feedback.

What makes a good code assertion?

The key to a good code assertion eval is you have to find a fixed structural approach to measuring the error in a way that doesn't require judgment. Examples:

Hallucination guard. For every quote in a story ChatGPT wrote, the eval searched the original transcript for the exact string, returning true only on a match.
Red-flag words. The Product Talk Interview Coach eval string-searched for a list of words (like "typically" or "usually") that signal a general question.
Node counts. For an AI-generated opportunity solution tree, the eval counts the nodes, checks how well they are distributed, and uses thresholds to define error categories.

They also matter when a later step must parse the LLM's output, like checking for valid JSON before it reaches the downstream step.

When should you use a code assertion instead of an LLM-as-a-Judge?

Always try code first. For every new error that needs an eval, look for a way to measure it with deterministic code. It's fast, cheap, and deterministic, which makes it easy to debug. Only when an eval truly requires judgment should you turn to an LLM-as-a-Judge eval.

Even crude checks can align well with human evaluation, so start with the simplest, most obvious eval. It might work better than you expect.

Learn more:

Related terms:

← Back to AI Glossary

Last Updated: September 3, 2026