Upstream Errors
What are upstream errors?
Upstream errors are errors that happen in earlier LLM calls. Most AI products aren't a single prompt. They're multi-step pipelines: one LLM call generates something, a later call builds on it, and the final output reflects every step along the way. That means an error you observe in the final output may not originate there—it may trace back to an earlier call. A common lesson from the AI evals community is that we should always tackle upstream errors first.
How do upstream errors confuse downstream measurement?
Upstream errors don't just degrade the final output—they can also make downstream errors hard to measure. When Teresa Torres built an LLM-as-a-judge to count missed sub-groupings in AI-generated opportunity solution trees, the judge kept flagging groupings that weren't actually missing. Digging into those false positives revealed the judge was getting confused by upstream errors: a poorly framed key moment, and opportunities that simply restated their parents without adding new information. When those upstream errors were removed, the judge correctly ruled there were no missed groupings. The inputs were good enough for customers—no one was complaining about them—but they weren't good enough for the judge to do its job well.
What makes tackling upstream errors first tricky?
The advice is right, but it isn't simple in practice, because fixes interact. In Teresa's case, fixing the upstream errors exacerbated the downstream error—each improvement on one side made the other side worse. Being able to measure all of the errors was what was key. Build an eval for every error category, upstream and downstream, so you can see the trade-offs while you work toward fixing them all.
Related terms:
Learn more:
Last Updated: September 16, 2026