Customer Feedback Eval
What is a customer feedback eval?
A customer feedback eval is a type of AI eval where the customer tells you if the response was good enough. It's one of the four common types of evals, alongside golden dataset evals, code assertions, and LLM-as-a-Judge evals. For customer-facing products, it's the ultimate judge. A code assertion or an LLM-as-a-Judge can only measure the definition of correctness you gave it. The customer tells you whether the response actually worked for them.
What counts as customer feedback in an eval?
The most direct form is an explicit rating: a thumbs up or thumbs down, or a simple score. But ratings aren't your only option. Other customer behaviors can act as a proxy for feedback. If a customer uses AI to generate an image and then regenerates it, that's feedback that the first image didn't work for them. If they changed the prompt before regenerating, those changes might carry details about what went wrong.
Edits are another rich source. At Vistaly, the plan is to use customers' edits to AI-generated interview snapshots and opportunity solution trees as feedback on what the LLM got wrong. This category is endless.
Why are customer feedback evals hard to make actionable?
It's difficult to get customers to tell you more than a thumbs up or thumbs down. A low rating tells you the response fell short, but not which error category was responsible, so you often have to infer what went wrong. That's why behavioral proxies like regenerations, prompt changes, and edits matter. They carry more detail about the specific error than a rating does, which makes them easier to turn into something you can fix.
Related terms:
Last Updated: September 1, 2026