Should we use the small option?

Test it before assuming not. Run your own twenty cases against the small and the large model and count the differences that matter. Organisations frequently find the gap is invisible on their actual work while the cost is several times lower, and that finding is worth an afternoon.

Where will we notice the difference?

On rare subjects, on problems needing several dependent steps, and on anything requiring the model to hold a lot in mind at once. If your work is mostly summarising, classifying and drafting in a familiar domain, the difference may not appear at all.

Can we distil a provider's model ourselves?

Technically it is well understood and contractually it is contested. Provider terms commonly restrict using their output to train competing models, how enforceable that is gets argued about, and it is not a question this page can settle. Read the terms and take advice before building anything on the assumption.

Glossary

Distillation

Distillation trains a smaller model to imitate a larger one, producing something that keeps much of the quality while costing a fraction as much to run and answering considerably faster.

In plain terms

An apprentice trained by watching an expert. The small model is not shown the world afresh; it is shown what the large model produces and taught to produce the same thing. It ends up much better than a small model trained the ordinary way, and not as good as the one it was copying, which is a trade most of your tasks will be happy with.

Why it matters

It is why the cheap, fast option in a provider's range is far better than its price suggests, and why running a capable model on modest hardware is possible at all. When a vendor offers a small model that performs unexpectedly well, this is usually the reason. The practical consequence for you is that reaching for the largest model by default often means paying several times over for a difference your own test would not detect.

How it works

The large model produces answers and the small one is trained to match them, which is a richer signal than training on the same original material would be. The large model has already done the hard work of resolving ambiguity, and the small one learns from those resolved answers rather than from the raw problem.

It compresses capability rather than knowledge, and the distinction shows in what survives. A distilled model typically keeps the manner, the format and much of the competence on common tasks, while thinning out on rare subjects and long reasoning, because those are where the large model's extra size was doing work.

It combines readily with the other size reductions, particularly storing the numbers at lower precision, and the combination is what lets a genuinely useful model run on a laptop. Each step gives up a little, and together they change what you can run on hardware you already own.

Distilling somebody else's model raises a contractual question rather than a technical one. Provider terms frequently restrict using their output to train competing models, the enforceability of such terms is argued about, and this page does not settle it. Anybody planning to do it should read the terms and take advice rather than assume.

Where the small model keeps up, and where it starts to thin

The useful thing about this technique is that its losses are not spread evenly, which means you can find out whether they affect you. Toward the left the small model is genuinely hard to distinguish from the large one it copied, and that is where the great majority of ordinary business work sits. Toward the right the extra size was doing something, and if your work lives there you will see it immediately. The reason this matters commercially is the default: reaching for the largest available model is the safe-feeling choice and frequently pays several times over for a difference nobody in the organisation could point to. An afternoon spent running your real cases at both ends settles the question properly, and usually settles it in favour of the cheaper option.

Seen in the wild

Run a small published model on your own machine and notice it is far better than its size would suggest, which is usually this.
Ollama
Compare a provider's small and large options on your own twenty cases through one interface, where the price difference is far larger than the quality difference on most of them.
OpenRouter
Load a compact model locally and try a rare subject and a long chain of reasoning, which is where the thinning shows.
LM Studio

Common misconceptions

People assume

A distilled model is just a worse one.

In fact

It is a smaller one that performs well above its size, which is a different thing. On the ordinary work most organisations do, summarising, classifying, extracting and drafting, the gap against the large model it copied is frequently undetectable, while the cost and the wait are several times lower.

People assume

The quality loss is spread evenly.

In fact

It tends to concentrate in rare subjects and long chains of reasoning, which is precisely where the extra size was earning its place. That is useful to you, because it means your own test set will find the gap if your work contains those cases and will find nothing if it does not.

Telling them apart

Distillation vs Quantisation

Distillation

A new, smaller model trained to imitate a larger one. Different model, different capability profile.

Quantisation

The same model with its numbers stored less precisely. Same model, smaller footprint, slight loss.

One rebuilds and the other compresses. They are routinely combined, and the compact model on a laptop is usually the result of both.

Questions

Should we use the small option?: Test it before assuming not. Run your own twenty cases against the small and the large model and count the differences that matter. Organisations frequently find the gap is invisible on their actual work while the cost is several times lower, and that finding is worth an afternoon.
Where will we notice the difference?: On rare subjects, on problems needing several dependent steps, and on anything requiring the model to hold a lot in mind at once. If your work is mostly summarising, classifying and drafting in a familiar domain, the difference may not appear at all.
Can we distil a provider's model ourselves?: Technically it is well understood and contractually it is contested. Provider terms commonly restrict using their output to train competing models, how enforceable that is gets argued about, and it is not a question this page can settle. Read the terms and take advice before building anything on the assumption.

Key takeaways

A smaller model is trained to imitate a larger one, keeping much of the quality at a fraction of the cost.
It learns from resolved answers rather than raw material, which is why it beats a small model trained ordinarily.
The loss tends to concentrate in rare subjects and long reasoning rather than spreading evenly.
It combines with lower-precision storage, and that combination is what runs a useful model on a laptop.
Distilling somebody else's model is a contractual question that is genuinely contested.

Tools that use this

Ollama
Small published models that punch above their size.
OpenRouter
Small against large on your own cases, with the price difference visible.
LM Studio
Locally, where rare subjects and long reasoning show the thinning.

Last checked July 2026

All glossary terms