Where does the effort actually go?

Into labelling, almost always. Modelling is usually the shorter part of one of these projects and organisations plan for the reverse, which is why they overrun in a direction that was entirely predictable before anybody started, and why the overrun surprises people anyway.

What do we do when labellers disagree?

Treat it as a finding rather than as a defect. Persistent disagreement means the category itself is ambiguous, and either the categories need redesigning or the system should be allowed to express uncertainty rather than being taught to pick one answer confidently.

What happens to cases nobody anticipated?

They are forced into whichever existing category fits least badly, because the system's world is exactly the set of outcomes somebody wrote down. Nothing in the design produces a category meaning this is something new, so novelty arrives disguised as a confident ordinary answer.

Glossary

Supervised learning

Training on examples where somebody has already written down the right answer, which is how most systems that sort or predict things get built.

In plain terms

Teaching by example, where each example comes with the answer attached. Somebody has to attach those answers, and everything the system ends up believing traces back to the judgements that person made while doing it.

Why it matters

Because the part everybody plans for is the modelling and the part that decides the outcome is the labelling. A system trained on inconsistent answers learns the inconsistency perfectly well, and no amount of technique recovers from it afterwards.

How it works

The shape is examples with answers attached. Given enough pairs of a case and its correct outcome, the system learns to produce the outcome for cases it has not seen, which is the whole idea and the reason it needs the answers in the first place.

Those answers are judgements somebody made. Every label carries whatever interpretation, mood, hurry or house convention its author brought, so the finished system reproduces the labellers rather than reproducing the truth.

Disagreement between labellers is information rather than noise. Where two people label the same case differently, the category is genuinely ambiguous, and a project that resolves the disagreement by picking one is teaching a system to be confident about something people cannot agree on.

Labelling is where the cost and the calendar go. Modelling is usually the shorter part of a supervised project, and organisations consistently plan for the reverse, which is why these projects overrun in a predictable direction.

The categories have to be decided before labelling starts, and revising them means relabelling. That makes the design of the categories the most consequential hour of the project and the one most often given to whoever is available.

Where the labels come from shapes what can be learned, and historical records are the usual source. Past decisions carry whatever the organisation was doing at the time, including the parts it has since decided were wrong, so training on its own history is how a practice gets preserved rather than examined.

It remains the right approach for a great many problems, which is worth saying plainly. Where the categories are stable, the examples are plentiful and the judgement is genuinely routine, a system trained this way is cheaper to run, easier to explain and more predictable than anything more elaborate.

What it cannot do is answer a question nobody has labelled. The system's world is exactly the set of outcomes somebody wrote down, so a case belonging to a category that was never anticipated is placed into whichever existing one fits least badly.

Where a supervised project spends its time

The third line on the right is the one worth protecting rather than eliminating. Arguing about the hard cases feels like an obstacle to getting on with the labelling, and it is the part of the exercise that produces the most durable value: it is where an organisation discovers that two of its teams have been categorising the same situation differently for years, or that a distinction everybody assumed was obvious dissolves under three examples. Those findings outlive the model. They change how the work is done regardless of whether the system is ever deployed, which is more than most machine learning projects can say. The practical implication is to run that argument early and deliberately, on a small sample, with the people who actually do the work, before anybody labels a thousand cases according to a scheme that will not survive contact with the awkward tenth of them.

Seen in the wild

Historical support tickets labelled by category to train a router.
Julius AI
Two colleagues labelling the same cases differently, which is the useful finding.
Notion AI
A classifier trained on labelled examples rather than prompted to judge.
Hugging Face

Common misconceptions

People assume

The model learns the truth.

In fact

It learns the labels. Whatever the people attaching answers believed, including their inconsistencies and their local conventions, is what the finished system reproduces, and it does so confidently.

People assume

Disagreement between labellers is a quality problem.

In fact

It is usually a signal that the category is ambiguous. Forcing a single answer teaches the system to be certain about something people genuinely cannot agree on, which produces confident output on exactly the cases that deserve hesitation.

Questions

Where does the effort actually go?: Into labelling, almost always. Modelling is usually the shorter part of one of these projects and organisations plan for the reverse, which is why they overrun in a direction that was entirely predictable before anybody started, and why the overrun surprises people anyway.
What do we do when labellers disagree?: Treat it as a finding rather than as a defect. Persistent disagreement means the category itself is ambiguous, and either the categories need redesigning or the system should be allowed to express uncertainty rather than being taught to pick one answer confidently.
What happens to cases nobody anticipated?: They are forced into whichever existing category fits least badly, because the system's world is exactly the set of outcomes somebody wrote down. Nothing in the design produces a category meaning this is something new, so novelty arrives disguised as a confident ordinary answer.