When is it worth using?

When the volume makes human review impossible and something would otherwise go unmeasured. It is at its best on comparisons, where the question is whether this version is better than the previous one rather than what absolute score this deserves. Relative judgements survive the known tendencies better than absolute ones.

Should the judge be the same model that produced the output?

Preferably not, and the reason is shared blind spots rather than any principle about self-marking. A model asked to assess its own kind of output will miss the same things it missed while producing it. Using a different model does not eliminate that, since training material overlaps, and it does reduce it.

Can the results be compared over time?

Only while the judge and the rubric are held still. Both are easy to change for good reasons, and either change moves every score, so a comparison spanning it is measuring the instrument rather than the system. Recording which judge and which rubric produced a number is what makes a series mean anything later.

What is the most common way this goes wrong?

Trusting the numbers before comparing them with people, usually because the scores looked plausible and the comparison is the expensive part. The failure is quiet: the numbers stay stable while quality moves, and everyone believes the system is being watched when what is being watched is the judge's preferences.

Glossary

LLM as judge

LLM as judge is using one model to score another model's output, which makes evaluation affordable at scale and introduces a measuring instrument that has to be checked against people first.

In plain terms

Getting a model to mark another model's homework. Reading a thousand answers to see whether quality slipped is work nobody has time for, so a model reads them instead and scores each one. It is fast and cheap, and it is a measurement made by the same kind of thing being measured, which is the part that needs thinking about.

Why it matters

Because without it most teams measure nothing. Human review does not scale past a sample, samples get skipped when everyone is busy, and the result is a system whose quality nobody has checked since launch. A judge makes continuous measurement affordable, and it does that by producing numbers that look authoritative and are only as good as the agreement behind them.

How it works

The judge is given the output and a statement of what good looks like, and returns a score or a verdict. That statement is doing most of the work: a vague one produces scores that track fluency rather than correctness, because fluency is what a model notices most easily. Nothing about the technique fixes an unclear idea of quality, which is usually the real obstacle.

It is checked by agreement with people rather than by inspection. A sample gets scored both ways, and the question is whether the judge and the humans reach the same verdicts on the same items. Until that has been done the scores are unvalidated, and a validated judge is one that has been compared, not one that has been carefully written.

Known tendencies are worth knowing before reading any result. Judges tend to reward answers that are longer, more confident and stylistically similar to their own output, none of which is the same as being right. These are systematic rather than random, which means they do not average out across a large sample and can move a whole set of scores in one direction.

Cost is the reason it exists and also the reason it drifts. Scoring every output with a strong model can cost more than the work being scored, so teams move to a cheaper judge, a sample rather than everything, or a shorter rubric. Each is reasonable and each changes the instrument, which means a series compared across such a change is measuring two different things and will show a step nobody caused.

It shows up in products as quality watching rather than as a feature anybody buys. An enterprise agent platform in this guide has one surface watching conversation quality and another running multivariate tests of agent behaviour, and an application framework pairs tracing with evaluation tooling. In both cases the scoring is infrastructure for a team operating a system, not something the end user sees.

What the score is, and what it is taken for

The gap between these columns opens quietly, because nothing about a judge announces that it is unvalidated. The scores arrive on a dashboard looking exactly like scores that have been checked, they move sensibly when the system changes, and they are stable enough to build a habit around. That plausibility is the problem: a judge that consistently rewards longer and more confident answers will produce a stable, credible-looking series that tracks verbosity rather than quality, and nothing in the series will look wrong. The only thing that separates the two columns is the comparison against human judgement on the same material, which is tedious, has to be repeated when the judge or the rubric changes, and is the single step most likely to be skipped when the numbers already look reasonable.

Seen in the wild

An agent platform with one surface watching conversation quality and another running multivariate tests of how the agent behaves.
Sierra
A framework pairing tracing with evaluation tooling, which is what turns debugging agent behaviour into a discipline rather than archaeology.
LangChain / LangGraph
Bringing your own model keys and watching usage directly, which is the same instinct applied to cost rather than to quality.
Relevance AI

Common misconceptions

People assume

The scores are objective because a machine produced them.

In fact

They are a second model's opinion, expressed as a number. The number is repeatable, which is genuinely useful and is not the same as being correct. What makes a score trustworthy is evidence that it agrees with human judgement on the same material, and that evidence has to be produced rather than assumed.

People assume

A stronger judge model removes the problem.

In fact

It usually helps and it does not change the shape. A stronger model still prefers length and confidence, still shares blind spots with the model it is scoring where both were trained on similar material, and still needs comparing against people. The improvement is in accuracy, not in whether validation is required.

Telling them apart

LLM as judge vs Evals

LLM as judge

One technique for producing the score.

Evals

The whole practice of measuring whether output is good enough.

Evals is the discipline and this is one instrument in it. Choosing the instrument before deciding what good means is the common order and the wrong one.

Questions

When is it worth using?: When the volume makes human review impossible and something would otherwise go unmeasured. It is at its best on comparisons, where the question is whether this version is better than the previous one rather than what absolute score this deserves. Relative judgements survive the known tendencies better than absolute ones.
Should the judge be the same model that produced the output?: Preferably not, and the reason is shared blind spots rather than any principle about self-marking. A model asked to assess its own kind of output will miss the same things it missed while producing it. Using a different model does not eliminate that, since training material overlaps, and it does reduce it.
Can the results be compared over time?: Only while the judge and the rubric are held still. Both are easy to change for good reasons, and either change moves every score, so a comparison spanning it is measuring the instrument rather than the system. Recording which judge and which rubric produced a number is what makes a series mean anything later.
What is the most common way this goes wrong?: Trusting the numbers before comparing them with people, usually because the scores looked plausible and the comparison is the expensive part. The failure is quiet: the numbers stay stable while quality moves, and everyone believes the system is being watched when what is being watched is the judge's preferences.

Key takeaways

A second model's opinion expressed as a number: repeatable, not automatically correct.
The statement of what good looks like does most of the work.
Validation means agreement with people on the same items, and it has to be produced.
Known tendencies favour length and confidence, and they are systematic rather than random.
Relative comparisons hold up better than absolute scores.

Tools that use this

Sierra
Conversation quality watched, and agent behaviour tested multivariately.
LangChain / LangGraph
Tracing paired with evaluation tooling for agent behaviour.
Relevance AI
Usage watched directly, the same instinct applied to cost.

Last checked July 2026

All glossary terms