Sign in Subscribe

Code Assertion

What is a code assertion?

A code assertion (also called a code-based assertion) is a type of AI eval strategy where you use traditional deterministic code to evaluate the quality of an LLM response. Instead of using complex evaluation methods, you write simple code that checks whether the response meets specific criteria.

Common examples include checking if the LLM is returning valid JSON, looking for specific keywords (like "typical" or "usually" to detect general questions), or validating that the response follows expected formatting rules.

Code assertions are one of three main eval strategies, alongside golden datasets and LLM-as-Judge evals.

Why are code assertions effective?

Code assertions can be surprisingly simple yet highly effective. Even crude checks—like looking for the presence or absence of specific terms—can align well with human evaluation.

For example, to detect whether an AI is suggesting unwanted general questions (like "Tell me about your typical morning routine"), you can create a code assertion that simply looks for keywords like "typically," "typical," "general," "generally," "usual," or "usually." This straightforward approach can work very effectively and match how humans would evaluate the same responses.

The lesson: Start with the simplest, most obvious eval. It might work better than you expect.

When should you use code assertions?

Code assertions are particularly important in workflows where a subsequent step might need to parse the LLM's output. For instance, if your application expects the LLM to return structured JSON that another function will process, a code assertion can verify the JSON is valid before it reaches that downstream step.

Code assertions are also useful for catching obvious errors quickly. They're simple to write—often taking just a minute with AI assistance—and they help you think through different ways your system could fail.

Learn more:

Related terms:

← Back to Ai Glossary