What should we measure before labelling at scale?

Agreement on a small shared sample. Give the same items to two people, compare, and read the disagreements as a report on your guidelines rather than on the people. That measurement costs a morning and routinely prevents a project built on a category nobody could define consistently.

Can AI just do the labelling now?

It can do a strong first pass, which changes where people spend their attention rather than removing them. The risk is that reviewers concentrate on cases the system flagged and never examine the ones it found easy, so its blind spots become the dataset's. Sampling the confident cases is the guard against that.

How many examples do we need?

Fewer than expected, if they are the right ones. Volume of obvious cases adds little, while the awkward and boundary examples carry most of the information, so effort spent finding disagreements beats effort spent adding more of the easy majority. Quality here is a statement about selection rather than about care.

Glossary

Data labelling

Data labelling is marking up examples so a system can learn from them, and the disagreement between the people doing it is more informative than the labels they produce.

In plain terms

Going through examples and marking what each one is, so something can learn the pattern. Which emails are complaints, which invoices are duplicates, which photographs contain a defect. It is slow, it is usually done by people, and the interesting part is what happens when two of them disagree about the same example.

Why it matters

Because disagreement is a measurement of the task rather than of the people. If two careful colleagues label the same items differently, the category itself is ambiguous, and nothing trained on those labels will be clearer than the instructions were. Discovering that early costs a morning; discovering it after training costs the project.

How it works

Agreement between labellers is the number worth collecting first. Giving the same sample to two people and comparing is cheap, and the result tells you whether the category is well enough defined to be learnable, which is a question about your definition rather than about anybody's diligence. Low agreement is a specification problem wearing a staffing costume.

Ambiguity in the guidelines propagates directly into behaviour. A boundary case decided differently on different days produces a system that is inconsistent in exactly the same places, so the effort spent making the instructions unambiguous is not preparation for the work, it is most of the work.

AI now does a large share of the first pass, which changes the shape rather than removing the job. A system proposes labels and people review, correct and adjudicate the difficult ones, which is far faster and concentrates human attention where the judgement genuinely is. It also risks entrenching the proposer's blind spots if nobody checks the cases it found easy.

The value sits in the awkward examples rather than the plentiful ones. Thousands of obvious cases teach very little, while the small pile that people argued about carries most of the information, so a labelling effort that samples randomly spends most of its money confirming what was already clear.

What two labellers disagreeing actually tells you

There is a particular kind of project failure that this measurement prevents, and it is worth describing because it looks like success until quite late. A team defines a category, labels several thousand examples, trains something, and finds the results maddeningly inconsistent on precisely the cases that matter commercially. The investigation goes to the model, then to the volume of data, then to the technique, and eventually somebody re-reads the guideline and discovers it never actually said what to do when an email was both a complaint and a renewal enquiry. Every labeller had been deciding that for themselves, differently, for months. The system learnt exactly what it was shown, which was a coin flip. Half a morning spent comparing two people's labels at the start would have surfaced it, and the reason that morning is so often skipped is that disagreement reads as a problem with people rather than as free information about the task.

Seen in the wild

The hub where open models and the datasets behind them are published together, so what something learnt from can be inspected rather than assumed.
Hugging Face
A database and workflow platform whose AI classifies records and extracts details against operational data already held in the workspace.
Airtable AI
A visual canvas where AI operations are first-class nodes and batches run over whole datasets, which is the first-pass shape at scale.
Gumloop

Common misconceptions

People assume

Disagreement means the labellers need better training.

In fact

Usually it means the category needs a better definition. Two careful people reading the same guideline and reaching different answers have found an ambiguity in the instruction, and no amount of diligence resolves a boundary nobody has drawn. Fix the definition and the agreement follows.

People assume

More labelled examples is straightforwardly better.

In fact

More of the obvious ones teaches almost nothing. The information concentrates in the cases people argued about, so a random sample spends most of its budget confirming what was already clear. Finding and labelling the awkward examples is worth far more than volume.

Telling them apart

Data labelling vs Training data

Data labelling

The work of marking examples up.

Training data

The material a model learnt from.

One is a process and the other is the result, and a great deal of training data carries no deliberate labelling at all.

Questions

What should we measure before labelling at scale?: Agreement on a small shared sample. Give the same items to two people, compare, and read the disagreements as a report on your guidelines rather than on the people. That measurement costs a morning and routinely prevents a project built on a category nobody could define consistently.
Can AI just do the labelling now?: It can do a strong first pass, which changes where people spend their attention rather than removing them. The risk is that reviewers concentrate on cases the system flagged and never examine the ones it found easy, so its blind spots become the dataset's. Sampling the confident cases is the guard against that.
How many examples do we need?: Fewer than expected, if they are the right ones. Volume of obvious cases adds little, while the awkward and boundary examples carry most of the information, so effort spent finding disagreements beats effort spent adding more of the easy majority. Quality here is a statement about selection rather than about care.

Key takeaways

Labeller disagreement measures the task, not the labellers.
Ambiguous guidelines produce inconsistency in exactly the same places.
AI does the first pass now, which concentrates rather than removes human judgement.
The awkward examples carry the information; the plentiful ones do not.