Semantic Task
What is a semantic task?
A semantic task is a task where the right answer depends on meaning rather than a fixed rule. It might have more than one right answer—or better and worse answers. Summarizing an interview, judging tone, and deciding whether a question is leading are all semantic tasks.
We turn to large language models for semantic tasks—that's their strength. But semantic tasks are harder to judge. It's rare that we can say this is right and this is wrong. More often, it's better or worse. There's a lot of gray.
How do semantic tasks differ from tasks with one right answer?
Some tasks have one correct output: classification (is this a business outcome or a product outcome?), factual answers (who was the first US President?), and routing (which skill best applies?). For these, a golden dataset eval works well: define the inputs and ideal outputs, then count how often the LLM matches.
Semantic tasks have no single ideal output. Ask an LLM to write a joke. Did it make you laugh? Is it clever? Is it appropriate for my kids? Each is a different dimension of correctness.
How do you define correctness for a semantic task?
Before you can measure how often an LLM gets it right, you have to define what a right answer looks like—usually on multiple dimensions. For an interview coach, that might mean: did it sort each question into the right section, did it avoid suggesting a leading question, and was the tone appropriate?
Don't outsource this work. Vendors offer built-in evals for conciseness or helpfulness, but correctness is context dependent. Defining what good looks like is the product team's job. Then measure each dimension with a code assertion or an LLM-as-a-judge eval.
Related terms:
Last Updated: September 1, 2026