Probabilistic
What does it mean that LLMs are probabilistic?
Traditional code is deterministic. You can look at the code and know exactly what you'll get every time. Large language models are probabilistic. That means their output can vary, even given the same input. Ask an LLM the same question twice and you might get two different answers.
If you write a function that adds two numbers, you can write a unit test that sends it 2 and 3 and checks that it returns 5. Write that test once and run it every time you change the code. If you build an LLM service to do the same thing, the first time you send it 2 and 3 it might return 5. The second time it might return 4.
Why don't traditional testing methods work for LLMs?
Unit tests assume the same input always produces the same output. That holds for code and breaks for LLMs. Testing an LLM once isn't adequate. It might work once and then break the next time.
This is why AI evals measure a rate rather than a pass/fail. With a unit test, we expect the code to always get it right. With an eval, we are measuring how often the LLM gets it right.
What does this mean for product teams?
You can't test your AI product or workflow once and trust it. Instead, run many inputs through it, count how often the output is correct, and use that error rate as your baseline. When you change a prompt, add context, or swap models, run the same inputs again and compare.
It's also why error analysis matters. You only learn what mistakes a model makes by looking at many outputs, not one.
Related terms:
Last Updated: September 1, 2026