What should go in it?

The awkward cases rather than the typical ones. Easy examples are answered correctly by anything, so a set drawn from ordinary work reports that everything is fine regardless of what has happened, which is worse than having no set at all.

When can we change it?

Ideally never, and where new material is genuinely needed, add a second set rather than editing the first one. The moment answers are revised, this month and last month stop being comparable, which was the whole reason for keeping the set in the first place.

What is the hardest part?

Agreeing what the right answers actually are. Two colleagues will often disagree about the correct handling of a difficult case, and settling that argument before anything is deployed is usually worth more than every measurement the set goes on to produce afterwards.

Glossary

Golden dataset

A fixed set of examples with agreed correct answers, kept unchanged so a change in quality shows up as a difference rather than an impression.

In plain terms

A small pile of examples where everybody has agreed what the right answer is, kept exactly as it is. When something changes, you run it again and see what moved, instead of relying on a sense that things feel worse than last month.

Why it matters

Because without one, every question about whether a system has got worse is answered by impression, and impressions are formed by whoever complained most recently. A fixed set turns an argument into a comparison that anybody can run.

How it works

Its entire value is that it does not change. The moment examples are added or answers revised, this run and last month's run are no longer comparable, and the one property you were paying for has gone.

That makes the temptation obvious and the discipline hard. When a system starts failing cases in the set, the quickest way to restore a good result is to decide those cases were unfair, and that decision is always available and always sounds reasonable.

What belongs in it is the awkward material. Cases that are easy get answered correctly by anything, so a set drawn from typical work will show no difference between a good system and a poor one and will keep reporting that everything is fine.

Agreeing the answers is most of the work and most of the value. Two colleagues frequently disagree about what the right answer is, and discovering that before deployment is worth more than the measurements the set later produces.

It is a comparison instrument rather than a grade. A score on it is not a statement about how good the system is; it is a number that means something when compared with the same number from before a change.

Running it costs almost nothing once it exists, which is what makes it worth building. A set small enough to be checked by hand can be rerun on every model change, every prompt revision and every supplier claim, and the whole exercise fits in an afternoon each time.

It has to be kept out of anything the system learns from. A set that finds its way into training material stops measuring anything, and this is easier to do accidentally than it sounds once documents are being gathered in bulk.

What happens when it starts failing

What makes this difficult is that the left-hand column is sometimes right. Occasionally an expected answer really was wrong, or a case really was ambiguous, and correcting it is the correct thing to do. The problem is that the moment to notice this is never the moment a score has just dropped, because that is exactly when everybody has a reason to find the case unfair. The workable discipline is procedural rather than heroic: changes to the set require somebody other than whoever is being measured to agree, and every change is dated and recorded, so that a series can be read with the amendments visible in it. That costs almost nothing and it makes the difference between a measurement anybody can trust and a number that has been gently adjusted towards comfort over eighteen months by people who each did something reasonable.

Seen in the wild

A fixed set of awkward questions rerun after a model version changes.
ChatGPT
Agreed answers written down by two colleagues who initially disagreed.
Notion AI
A held-aside collection kept well away from anything used for training.
Hugging Face

Common misconceptions

People assume

It should be kept up to date.

In fact

Updating it destroys the comparison it exists to provide. If new cases are genuinely needed, add a second set and keep the original untouched, so that the historical series remains readable.

People assume

A high score means the system is good.

In fact

It means the system handles those examples. The number is meaningful in comparison with the same number from before a change, and as an absolute grade it says only that somebody chose examples this system can pass.

Questions

What should go in it?: The awkward cases rather than the typical ones. Easy examples are answered correctly by anything, so a set drawn from ordinary work reports that everything is fine regardless of what has happened, which is worse than having no set at all.
When can we change it?: Ideally never, and where new material is genuinely needed, add a second set rather than editing the first one. The moment answers are revised, this month and last month stop being comparable, which was the whole reason for keeping the set in the first place.
What is the hardest part?: Agreeing what the right answers actually are. Two colleagues will often disagree about the correct handling of a difficult case, and settling that argument before anything is deployed is usually worth more than every measurement the set goes on to produce afterwards.