When is it clearly the right choice?

When the work is narrow and repetitive, when volume is high enough that per-item cost dominates, or when nothing may leave the building. The last of those is decisive rather than economic: a hosted model is not available at any price if the data cannot travel, so the comparison never happens.

What do we actually lose?

Headroom on the hardest cases. Ambiguous instructions and long multi-step problems are where the difference is real and visible; routine drafting, classification and extraction are where it very nearly disappears. Testing your hardest recurring case rather than a representative one settles it quickly.

Is this the same as running something locally?

They travel together and are not the same claim. Compact size is what makes local running practical, and a small model can equally be served from somewhere central for speed and cost. The size is a property of the model; where it runs is a separate decision that the size makes possible.

Glossary

Small language model (SLM)

A small language model is compact by design rather than by shortfall, and the reason to choose one is frequently that it can run somewhere a large one cannot go at all.

In plain terms

A deliberately compact one. It gives up some breadth and returns speed, low cost and the ability to run on ordinary hardware, including hardware that never touches a network. For a narrow, repetitive job that is frequently a better fit than something enormous, and for some jobs it is the only thing that fits at all.

Why it matters

Because the decisive advantage is often eligibility rather than economy. A compact model runs on a laptop, on a device in a factory, inside a hospital's own network or on a phone with no connection, and in each of those situations the question is not which model performs best but which models can be there at all.

How it works

Compactness is a design goal pursued from the start rather than a shortfall against something larger. Training material is chosen more carefully, the architecture is built for the size, and the result is a system that does a narrower range of things well instead of a wide range adequately, which is a different product rather than a lesser one.

What it gives up is breadth and depth on the hardest problems. Ambiguous instructions, long multi-step reasoning and unusual requests are where the gap opens, and summarising, classifying, extracting and routine drafting are where it closes to very little, which happens to describe most of the volume in an ordinary business.

Speed and cost change the shape of what is affordable, not merely the bill. Something cheap and fast enough to run over every record rather than a sample makes a different class of work possible, so the saving is frequently spent on doing far more rather than on doing the same for less.

Running where nothing else can is the argument that decides many deployments. No network dependency, no data leaving the building, and a response that does not vary with somebody else's capacity are properties a hosted frontier model cannot offer at any price, which is why regulated and disconnected settings reach for this first.

Two reasons to choose one, and only one of them is money

The distinction is worth drawing because the two cases lead to different conversations. An economic argument invites a counter-argument: somebody points out that the larger model is better, a budget is found, and the compact option loses on the merits it was never claiming. An eligibility argument does not work that way. A factory floor with no reliable connection, a hospital network that material may not leave, a device that must answer while offline: in each of these the large model is not an expensive option, it is absent from the list. Teams that lead with cost frequently find themselves defending a position they did not need to defend, when the honest framing was that the requirement had already eliminated everything else. It is worth establishing which argument you are actually making before the meeting, because only one of them can be lost.

Seen in the wild

A local runner that loads a model and serves it from your own machine, where the practical ceiling is the hardware rather than the ambition.
Ollama
A privacy-first desktop assistant that is fully offline by default with no account required, bundling engine and interface into one install.
Jan
An open-weight family published in several sizes, so the same line can be served on a server or on a laptop as the work requires.
Llama (Meta)

Common misconceptions

People assume

It is a cheap substitute for a proper model.

In fact

It is a different design decision with its own advantages, chief among them being able to run where nothing large can. Treating it as a compromise misses that for disconnected, regulated or very high-volume work the large model is not an option being declined, it is ineligible.

People assume

The gap shows up everywhere.

In fact

It concentrates on ambiguity and on long multi-step work. Summarising, classifying, extracting and routine drafting are handled comparably, and those account for most of the volume in an ordinary business, which is why the substitution works better in practice than a benchmark comparison suggests.

Telling them apart

Small language model vs Frontier model

Small language model

Compact by design; can run where nothing large can.

Frontier model

The most capable available, at the price that implies.

These are ends of the same trade, and the common arrangement is one of each rather than a choice between them.

Questions

When is it clearly the right choice?: When the work is narrow and repetitive, when volume is high enough that per-item cost dominates, or when nothing may leave the building. The last of those is decisive rather than economic: a hosted model is not available at any price if the data cannot travel, so the comparison never happens.
What do we actually lose?: Headroom on the hardest cases. Ambiguous instructions and long multi-step problems are where the difference is real and visible; routine drafting, classification and extraction are where it very nearly disappears. Testing your hardest recurring case rather than a representative one settles it quickly.
Is this the same as running something locally?: They travel together and are not the same claim. Compact size is what makes local running practical, and a small model can equally be served from somewhere central for speed and cost. The size is a property of the model; where it runs is a separate decision that the size makes possible.

Key takeaways

Compact by design, not by shortfall; a different product rather than a lesser one.
The gap concentrates on ambiguity and long multi-step work.
Cheap and fast changes what is affordable to attempt, not just the bill.
Where nothing large is eligible, this is the only comparison that matters.