Why is this harder than judging an answer?

Because the thing being judged is an outcome rather than a text. Checking it usually means looking at what changed: the record, the message, the file. That is closer to integration testing than to marking, and it cannot be run at the volume that answer-level checking can.

Does the path matter or only the result?

Both, and reading only the result hides real differences. Arriving correctly after twenty expensive turns that touched things they should not have is not the same outcome as arriving in three, even though a check on the end state scores them identically.

How should a passing result be read?

As one observation rather than a verdict. Because the route is decided during the run, the same case can behave differently next time. Anything used to decide whether something is ready needs a rate across repeated attempts, not a single green result.

Glossary

Agent evaluation

Agent evaluation asks whether a whole multi-step job was actually completed, which is a harder question than whether any single answer in it was any good.

In plain terms

Checking whether the thing actually did the job. Not whether each step looked sensible, which is easy to assess and not the question, but whether the outcome is what was wanted. Those come apart more often than people expect: a run made entirely of reasonable steps can end somewhere useless.

Why it matters

Because every step can be defensible and the result still wrong, so step-by-step review gives false comfort. The only assessment that answers the question compares the end state against what was asked for, and that is expensive, slow and requires somebody to have written down what success meant.

How it works

The unit is a finished job rather than an answer, which changes what can be automated. Judging a paragraph is a comparison anybody can run at volume. Judging whether a task was completed usually means inspecting the world afterwards: was the record updated, was the right thing sent, does the code pass its tests. That is closer to integration testing than to marking.

The path matters as well as the outcome, which is the part unique to this. A job can arrive at the right place having taken twenty turns, spent heavily and touched things it should not have. That is a different result from arriving in three, and any assessment reading only the final state calls them equal.

Repeatability is poor by construction, which weakens what a passing test proves. The route is decided during the run, so the same task attempted twice can take different paths, and a case that passed may fail on a later attempt without anything having changed. Measurement here is a distribution rather than a verdict.

The tooling has moved from an afterthought to a stated part of the platform. Frameworks now pair tracing with evaluation, and one describes that combination as what turned debugging behaviour into a discipline rather than archaeology. Enterprise products carry the same idea as surfaces that watch quality and run tests of behaviour.

Two things called testing here

The pull toward the left column is strong and understandable. It is cheap, it produces a stable number, and it can run on every change, which are exactly the properties that make a measurement usable. It also does not answer the question. A system can produce impeccable individual outputs at every step and fail to complete the job, and the left column will report health throughout, because nothing in it ever looks at the world. What makes the right column hard is not technical: it requires somebody to have written down what completion means for this task, in enough detail that a machine can check it. Where that has been done, as in the well-specified backlogs these tools are reported to handle best, evaluation becomes tractable almost immediately. Where it has not, teams fall back on reading traces and forming an impression, which is diagnosis wearing the costume of measurement.

Seen in the wild

A framework pairing tracing with evaluation tooling, described as what turned debugging behaviour into a discipline rather than archaeology.
LangChain / LangGraph
A platform with one surface watching conversation quality and another running multivariate tests of how the thing behaves.
Sierra
An autonomous engineer whose sweet spot is the well-defined backlog, where completion can be checked because the task was specified.
Devin Cloud

Common misconceptions

People assume

If each step looks right, the run was good.

In fact

Steps are generated the same way whether the approach is sound or not, so a coherent sequence is equally consistent with converging and with wandering. Reading the trace is genuinely useful for diagnosis and it is not an assessment, because the thing being assessed is the outcome.

People assume

A passing test means it works.

In fact

It means it worked that time. The route is chosen during the run, so the same case can fail on a later attempt with nothing changed. That makes a single pass much weaker evidence here than for ordinary software, and it is why results are read as a rate rather than as a verdict.

Telling them apart

Agent evaluation vs Evals

Agent evaluation

Was the whole job completed, and at what cost along the way.

Evals

Was this output good enough.

One inspects the world afterwards; the other compares a string. The first cannot be run at the volume of the second.

Questions

Why is this harder than judging an answer?: Because the thing being judged is an outcome rather than a text. Checking it usually means looking at what changed: the record, the message, the file. That is closer to integration testing than to marking, and it cannot be run at the volume that answer-level checking can.
Does the path matter or only the result?: Both, and reading only the result hides real differences. Arriving correctly after twenty expensive turns that touched things they should not have is not the same outcome as arriving in three, even though a check on the end state scores them identically.
How should a passing result be read?: As one observation rather than a verdict. Because the route is decided during the run, the same case can behave differently next time. Anything used to decide whether something is ready needs a rate across repeated attempts, not a single green result.

Key takeaways

The unit is a finished job, so checking usually means inspecting what changed.
A coherent sequence of steps is equally consistent with converging and wandering.
The path matters: the same outcome reached expensively is a different result.
Repeatability is poor by construction, so results are a rate rather than a verdict.

Tools that use this

LangChain / LangGraph
Tracing paired with evaluation, turning debugging into a discipline.
Sierra
Quality watched and behaviour tested multivariately as product surfaces.
Devin Cloud
The well-defined backlog, where completion can actually be checked.