Which version should we download?

Start at a moderate reduction, run your own cases, and move to a smaller one only while those still pass. That takes an afternoon and produces an answer specific to your work, which is worth considerably more than the general advice attached to any particular suffix on a download page.

How much memory will we save?

Roughly in proportion to the precision given up, so halving the bits per number approximately halves what the model needs to run. That arithmetic is what decides whether a given model fits your machine, and it is the number to work from when choosing hardware.

Does it affect speed as well as size?

Usually yes and favourably, because less data has to be moved around while the model runs. On hardware that was struggling to fit the model at all, the improvement can be dramatic rather than marginal, since the alternative was not running it.

Glossary

Quantisation

Quantisation, spelled quantization in most tooling, stores a model's numbers at lower precision so the whole thing takes up far less room and runs on smaller hardware, in exchange for a little accuracy.

In plain terms

Rounding, applied to everything at once. A model is millions of numbers, and storing each to fewer decimal places makes the whole thing dramatically smaller without changing what it is. Some precision is lost and the model still behaves like itself, which is why a version that would not fit on your laptop suddenly does.

Why it matters

It is the reason running a capable model on your own machine is possible at all. Without it, anything worth using would need hardware most organisations do not have; with it, the same model fits in a fraction of the memory and answers faster. Anybody looking at local or on-premises options is choosing between versions of this, usually presented as a set of confusing suffixes on a download page.

How it works

The numbers making up the model are stored with fewer bits each, which shrinks the file and reduces the memory needed to run it, roughly in proportion. The saving is arithmetic rather than clever, and the whole technique is the observation that most of those decimal places were not earning their place.

Quality falls gradually rather than suddenly as precision drops, up to a point where it falls off sharply. The useful consequence is that moderate reductions are close to free and aggressive ones are not, so the practical approach is starting at a moderate setting and moving down only while your own tests still pass.

Not every part of a model is equally tolerant, so good methods reduce precision unevenly, keeping more of it where the model is sensitive. That is why two versions of the same model at nominally the same size can behave differently, and why the method matters as much as the number in the name.

The losses show up first in the same places distillation's do: rare subjects, long chains of reasoning, and anything near the edge of what the model could do anyway. On ordinary work the difference is frequently undetectable, which is a claim your own cases can check in an afternoon.

How far down to go, and what it costs

The shape of this line is what makes the technique worth using: the early part of it is nearly free and the far end is not. Moving from full precision to a moderate reduction typically costs almost nothing you could detect on ordinary work while halving what the model needs to run, which is why practically every locally run model sits somewhere in that region. Pushing further keeps saving memory and starts costing quality on the harder tasks, and the far end degrades sharply enough that the model stops being worth running. Because the curve is shaped this way rather than being a straight trade, the sensible approach is not choosing a point in advance. It is starting moderate, running your own cases, and moving down only while they keep passing.

Seen in the wild

Download two versions of one model at different precisions and run your own questions against both, which is the only comparison that answers anything.
Ollama
Watch how much memory a model needs as you change versions locally, which makes the arithmetic concrete rather than abstract.
LM Studio
Look at a published model family offering several sizes and precisions, which is the menu everybody choosing local hardware ends up reading.
Qwen

Common misconceptions

People assume

It makes the model worse in general.

In fact

It makes it slightly worse in specific places while leaving most behaviour intact. At moderate settings the difference on ordinary tasks is often undetectable, which is why nearly every model anybody runs locally is a reduced-precision version and why that is not treated as a compromise.

People assume

The number in the name is the whole story.

In fact

The method matters too, because good approaches reduce precision unevenly and keep more of it where the model is sensitive. Two versions labelled with the same size can behave differently for that reason, which is a further argument for testing rather than reading the label.

Telling them apart

Quantisation vs Distillation

Quantisation

The same model, stored less precisely. Nothing was retrained, and the capability profile is broadly what it was.

Distillation

A new, smaller model trained to imitate a larger one. Different model, and a different profile of what it kept.

Compression against rebuilding. The compact model running well on a laptop is usually both, applied one after the other.

Questions

Which version should we download?: Start at a moderate reduction, run your own cases, and move to a smaller one only while those still pass. That takes an afternoon and produces an answer specific to your work, which is worth considerably more than the general advice attached to any particular suffix on a download page.
How much memory will we save?: Roughly in proportion to the precision given up, so halving the bits per number approximately halves what the model needs to run. That arithmetic is what decides whether a given model fits your machine, and it is the number to work from when choosing hardware.
Does it affect speed as well as size?: Usually yes and favourably, because less data has to be moved around while the model runs. On hardware that was struggling to fit the model at all, the improvement can be dramatic rather than marginal, since the alternative was not running it.

Key takeaways

It stores the model's numbers less precisely, shrinking memory and file size roughly in proportion.
Quality falls gradually and then sharply, so moderate reductions are close to free.
Good methods reduce precision unevenly, which is why the method matters as much as the number.
Losses appear first on rare subjects and long reasoning, as they do with distillation.
It is why running a capable model on your own machine is possible at all.

Tools that use this

Ollama
Two precisions of one model, side by side on your own questions.
LM Studio
Memory use changing as you swap versions, which makes the arithmetic concrete.
Qwen
A published family offering several sizes and precisions.

Last checked July 2026

All glossary terms