Evalsevaluations

Because without them nobody can answer the two questions that decide whether AI work continues: is this good enough, and is it still as good as it was. Generative output has no single correct answer, so the accuracy figure that would settle it does not exist, and in its absence the loudest recent experience wins. That is how a system degrades for months without anybody noticing, and how two teams end up disagreeing about a tool with no way of resolving it except by taking turns to be confident.

Start from real cases, not invented ones. Take examples from actual work, including the awkward ones people remember, because the value of the set comes entirely from how representative it is. Twenty genuine cases beat two hundred synthetic ones, and assembling them is usually a morning rather than a project.

Write down what good looks like for each. This is the hard part and doing it is most of the benefit. It forces the vague standard in somebody's head into something another person could apply, and teams frequently discover at this point that they disagree about the requirement rather than about the tool.

Score consistently, and accept that consistency matters more than sophistication. A human reading twenty outputs against a short rubric produces something useful. An elaborate scoring scheme applied inconsistently produces numbers that look rigorous and cannot be compared with last month's, which is worse than the simple version.

Run them when anything changes, which is more often than people expect. A new model version, a changed instruction, a different supplier, a product update you did not request: all of these can move quality, and the last of them can arrive with no announcement at all. The whole point of a fixed set is to detect that rather than to be surprised by a complaint.

Use a model as a scorer only once you have checked it against human judgement. Having one model grade another is how this scales, and it is only trustworthy where somebody has confirmed the scores agree with people on a sample. Skipping that check produces a measurement apparatus that is confidently wrong in a way nothing downstream will catch.

Keep the set fixed and add to it deliberately, because a set that drifts cannot be compared over time. When a genuine new failure appears, add it as a case; that is how the set improves. Quietly dropping cases that started failing is the failure mode, and it is easy to do without noticing you have done it.

Somebody has to own rerunning them, and naming that person is most of what makes this stick. The set itself is cheap to build and the discipline is what decays: a case set nobody reruns is a document rather than a control, and it fails silently because a set that is never run never reports a problem. Attaching the rerun to something that already happens, a release, a monthly review, a scheduled job, survives changes of personnel far better than an intention does.

The cases are worth sharing with the vendor, which most buyers never think to do. A supplier shown twenty real examples of where their product falls short has something concrete to work with, and the conversation stops being about impressions. It also tells you a great deal about the vendor, because the useful ones engage with the cases and the rest explain why your examples are unrepresentative.

Separate the two questions the set answers. Is it good enough to ship is a threshold question asked once; is it still as good is a monitoring question asked forever. The same cases serve both, but the second only works if somebody actually reruns them, which is the part that lapses first.

• Keep twenty real support questions with model answers, and rerun them whenever the underlying assistant changes version.

  ChatGPT

• Run the same case set against several models through one interface to compare quality on your work rather than on a public benchmark.

  OpenRouter

• Test a retrieval setup by asking questions whose answers you know are in the documents, and questions whose answers are deliberately not.

  NotebookLM

• Wire the case set into an automation so it runs on a schedule and reports rather than waiting for somebody to remember.

  n8n

What does a first set look like?

Twenty real cases drawn from actual work, each with a short note on what an acceptable answer contains, held somewhere the team can see. Run them, read the outputs, and write down the score. That is enough to detect change and to have an argument about quality that refers to evidence rather than to recollection.

How often should they run?

Whenever something changes and on a slow regular cadence otherwise. Changes include the obvious ones you make and the invisible ones you do not, such as a purchased tool quietly moving to a different model. The regular run is what catches the second category, which is the category nobody gets told about.

Can a model grade the outputs?

Yes, and it is how this scales past a handful of cases, but only after somebody has checked that its scores agree with human judgement on a sample. Without that check you have a measurement system whose errors are invisible, which is more dangerous than having no measurement because it carries false authority.

What should we do when a case starts failing?

Investigate before adjusting, because the interesting information is why. Something changed: the model, the instructions, the material or the expectation. Quietly dropping the case or loosening the standard removes the signal and is the single easiest way to end up with a set that always passes and means nothing.

Do these apply to bought tools, or only to things we build?

Both, and buyers under-use them. A purchased tool can change what it runs on without telling you, so the same fixed case set is how you find out whether an update helped or hurt. It is also the most credible thing to bring to a vendor conversation, because it describes your work rather than their demonstration.

How do these relate to a proof of concept?

A proof of concept asks whether the thing can work at all; evals are how you answer that with evidence, and then keep answering it afterwards. The case set assembled to judge a proof of concept is usually the seed of the permanent set, which is a good reason to build it properly the first time.

• Evals are real cases plus a written standard, rerun whenever something changes.
• Without them, quality is decided by whoever tried the tool most recently.
• Writing down what good looks like is the hard part and most of the benefit.
• A model can score outputs once its scores have been checked against people.
• A set that quietly loses its failing cases stops measuring anything.

• 

  ChatGPT

  Rerunning a fixed case set when the underlying assistant changes version.

• 

  OpenRouter

  The same cases against several models, compared on your work.

• 

  NotebookLM

  Testing retrieval with answers known to be present and known to be absent.