How can training work without anybody writing answers?

Because the material contains them. Hiding part of a passage and asking what was there produces a question and its correct answer together, so every sentence anybody has ever written is a supply of exercises that cost nothing to create.

Why does a model trained this way still need more work?

Because it has absorbed how language fits together without ever being told what to do with any of it. Useful behaviour comes from the stages that follow, which is why two systems built on the same foundation can end up behaving very differently indeed.

Glossary

Self-supervised learning

Training where the material provides its own answers, such as hiding part of a passage and scoring the guess, so no one has to write labels.

In plain terms

Getting the material to mark its own homework. Hide the next word, ask for a guess, compare with what was actually there. No person has to write down the right answer, which is why this can run over more text than anybody could ever label.

Why it matters

Because it removed the constraint that had bounded this work: somebody had to write the answers. Once the material supplies them, the amount available stops being limited by how many people you can pay, and everything about the field's shape follows from that.

How it works

The answer is constructed from the material rather than attached to it. Concealing part of a passage and asking for the missing piece produces a question and its correct answer at once, from text nobody prepared for the purpose.

That removes the bottleneck entirely. Labelling costs money per example and caps how much material can be used; this costs nothing per example, so the practical limit becomes how much text exists and how much computation you can afford.

The consequence is that nobody chose what was taught. A labelled collection reflects decisions somebody made, badly or well; material gathered at scale reflects whatever happened to be written down, including things nobody would have selected.

There is no equivalent of a labelling guideline. In supervised work you can improve the instructions given to labellers and relabel; here the closest equivalents act on the material or on later stages, and neither is as direct as changing what an answer is supposed to be.

It teaches structure rather than tasks. Predicting missing pieces produces a system that has absorbed how language fits together, which is why what comes out of this stage is a general capability rather than something that does a particular job.

Which is why a later stage is always needed. A system trained this way knows a great deal and has not been told what to do with any of it, so useful behaviour comes from what is done afterwards rather than from this stage alone.

Who decided what the system learned

It is tempting to read the right-hand column as strictly worse, and that is not the argument. The left-hand approach caps what can be learned at whatever a paid group of people could get through, which is a severe limit and the reason the field looked the way it did for a long time. Removing that cap is what produced systems general enough to be worth a glossary entry at all. The honest description of the trade is that one approach has a flaw you can name, inspect and fix by rewriting the instructions and doing the work again, and the other has a flaw distributed across more material than anybody will ever read. Both are real. What follows practically is modest and worth holding onto: a system built this way is not neutral because no one labelled it, and the absence of a labeller is not the absence of a point of view. It is the absence of anybody who could be asked what they meant.

Seen in the wild

A general model that knows a great deal and has to be told what job to do.
Llama (Meta)
Open-weight models whose published training material can be read about.
Hugging Face
The contrast: a small labelled collection assembled by hand for one task.
Julius AI

Common misconceptions

People assume

No labels means no human judgement involved.

In fact

The judgement moved rather than disappearing. Nobody labels individual examples, and decisions about what material to gather, what to exclude and what to weight are still judgements, made once and applied to everything.

People assume

It produces a system that can do a job.

In fact

It produces one that has absorbed structure and has not been told what to do. Everything about behaving usefully comes from later stages, which is why the same base can become tools that behave quite differently.

Questions

How can training work without anybody writing answers?: Because the material contains them. Hiding part of a passage and asking what was there produces a question and its correct answer together, so every sentence anybody has ever written is a supply of exercises that cost nothing to create.
What is the cost of removing the labelling step?: Nobody chose what was taught. A labelled collection reflects somebody's decisions, well or badly made; material gathered at scale reflects whatever happened to have been written down, and there is no equivalent of a labelling guideline you could rewrite to correct it.
Why does a model trained this way still need more work?: Because it has absorbed how language fits together without ever being told what to do with any of it. Useful behaviour comes from the stages that follow, which is why two systems built on the same foundation can end up behaving very differently indeed.