How do we know if it has happened?

Test on material from a genuinely different source: a later period, another team, another kind of case. A split taken from one collection cannot answer the question at all, because both halves carry the same peculiarities and the score reflects memory rather than learning.

Why is it so common in small projects?

Because small collections are what an organisation can realistically assemble, and small collections are where memorising is both easiest and hardest to spot. The project succeeds by every measure available to it and fails when it meets the wider world.

Would more data have prevented it?

Only if the extra data were different rather than merely more. Adding cases that resemble the ones already there reinforces exactly the same peculiarities, so variety across the situations the system will really encounter matters considerably more than sheer volume does.

Glossary

Overfitting

When a model learns its examples so closely that it does well on them and worse on anything it has not seen before.

In plain terms

Learning the examples instead of the lesson. The system looks excellent on everything it was shown and on anything closely resembling it, and disappoints the moment it meets something genuinely new, which is what you built it for.

Why it matters

Because it does not announce itself as a failure. The measurements look good, the project is declared a success, and the disappointment arrives weeks later in production, where it is usually attributed to the material being messier than expected.

How it works

The system is rewarded for matching the examples, and memorising them matches perfectly. Nothing in the training itself distinguishes a general rule from a well-remembered list, so both look like success while the training runs.

The classic tell is excellent results on held-back examples that came from the same batch. Splitting one collection into a training part and a testing part checks memory rather than generalisation, because both halves share whatever quirks that collection has.

Any peculiarity of the collection becomes part of what was learned. If every example was written by the same team, in one period, about one kind of case, then those regularities are as learnable as the thing you meant to teach, and far easier.

It is the quiet failure mode of a small fine-tuning project. Small collections are the ones organisations can realistically assemble, and small collections are exactly where memorising is easiest and hardest to detect.

The check that works is material from a genuinely different source. Examples from a later period, a different team or a different kind of case are the ones that show whether anything general was learned, and they are the ones nobody has ready.

It also has a milder and more common cousin that is worth naming: a system tuned repeatedly against the same evaluation set gradually learns that set, even when nobody trained on it directly. Each round of adjusting to a score is a small transfer of the answers into the design.

More examples help less than more varied ones. A thousand near-identical cases add little that a hundred did not; a hundred cases spread across the situations the system will actually meet are worth considerably more than either.

Two ways to hold back a test set

It is worth being fair about why the left-hand column dominates. Splitting a collection is a single line of code, it needs no additional material, and it is what every tutorial demonstrates because it is the version that can be shown in isolation. Assembling a genuinely independent sample is an errand: somebody has to find cases from a different period or a different team, confirm they are comparable, and hold them back untouched, which takes days and produces a worse-looking result at the end of it. Nobody is rewarded for that. The argument for doing it anyway is narrow and strong: the lower number is the one that predicts what happens in production, and finding it before deployment costs an errand while finding it afterwards costs the project's credibility. The practical version is to set aside that independent sample at the very start, before anybody has seen a score, because after the first good result nobody wants to go looking for the material that will spoil it.

Seen in the wild

A fine-tuned classifier that scores highly on its own test split and disappoints in use.
Hugging Face
A model taught from one team's historical cases, applied to another team's work.
Julius AI
A held-back sample drawn from a later period, which is the check that catches it.
vLLM

Common misconceptions

People assume

Good test scores mean it generalises.

In fact

Not if the test material came from the same collection. Both halves share whatever quirks that collection has, so a high score can be measuring how well the system remembers a batch rather than whether it learned anything.

People assume

More training examples is the fix.

In fact

More varied examples is the fix. A thousand near-identical cases teach little that a hundred did not, whereas a hundred spread across the situations the system will actually meet is a meaningfully different exercise.

Questions

How do we know if it has happened?: Test on material from a genuinely different source: a later period, another team, another kind of case. A split taken from one collection cannot answer the question at all, because both halves carry the same peculiarities and the score reflects memory rather than learning.
Why is it so common in small projects?: Because small collections are what an organisation can realistically assemble, and small collections are where memorising is both easiest and hardest to spot. The project succeeds by every measure available to it and fails when it meets the wider world.
Would more data have prevented it?: Only if the extra data were different rather than merely more. Adding cases that resemble the ones already there reinforces exactly the same peculiarities, so variety across the situations the system will really encounter matters considerably more than sheer volume does.