How is the routing decision actually made?

By rule, by a cheap classifier, or by escalating after a failed attempt. Rules are the most predictable and capture most of the saving; escalation is the most robust where failure is detectable; a classifier sits between and introduces a judgement that can itself be wrong. Most working deployments use rules by task type.

When is it not worth doing?

At low volume, where the classifier calls, the failed cheap attempts and the maintenance exceed the saving. It is also poor value where nearly all the work is genuinely hard, since the cheap path is rarely taken. Choosing the right model per workflow by hand achieves most of it with none of the machinery.

Quality falling without anybody noticing. A misrouted request returns a worse answer rather than an error, and no one downstream knows which model produced it. Sampling the cheap path's output periodically is the check that keeps the arrangement honest, and it is the first thing dropped once it appears to be working.

Does it help with reliability as well?

It can, since a system that already knows how to send a request elsewhere can send it elsewhere when a provider is unavailable. That is a different mechanism from cost routing and needs deliberate arrangement, but the two share the plumbing and building the first makes the second considerably cheaper.

Glossary

Model routing

Model routing sends each request to whichever model suits it, so the capable expensive one handles the hard cases and something cheaper handles the rest.

In plain terms

Most of the work an organisation sends to a model is straightforward, and a small proportion is genuinely hard. Paying the top rate for all of it is wasteful, and using something cheap for all of it produces bad answers on the cases that mattered. Routing is the attempt to tell the two apart automatically and send each to the right place.

Why it matters

Because the spread between what models charge for the same request is wide enough that the mix matters more than the negotiation. An organisation that sends everything to the most capable model available is usually paying a large multiple for the majority of its work, and the majority of its work would have been handled adequately by something cheaper. The saving is not marginal, which is why the idea is worth the complexity it introduces.

How it works

Something has to classify the request before the work is done, and that is the hard part rather than the sending. The classifier can be a rule about the type of task, a cheap model asked to judge difficulty, or a first attempt that is escalated when it goes badly. Each has a different failure mode and each costs something itself, which has to come out of the saving.

Escalation on failure is the most robust pattern and the least clever. Try the cheap model, detect that the answer is inadequate, and retry with the capable one. It only pays where the cheap model succeeds often enough that the retries are rare, and it depends entirely on being able to tell that an answer was bad, which is easy for a malformed result and hard for a plausible wrong one.

Routing by task type is simpler and more predictable, and it is where most real deployments end up. Classification, extraction and short summarising go to the cheaper model by rule; drafting, analysis and anything with judgement in it go to the capable one. It captures most of the available saving without needing a classifier that could itself be wrong.

The saving has to survive the routing overhead, which is where enthusiastic implementations disappoint. A classifier is itself a call, a failed cheap attempt is paid for as well as the retry, and the engineering has to be maintained as models change underneath. Where the volume is small the arrangement usually costs more than it returns.

The quiet risk is that quality falls invisibly. A misrouted request produces a worse answer rather than an error, and nobody downstream knows which model answered them. Sampling the cheap path's output periodically is what keeps the saving honest, and it is the step most likely to be skipped once the arrangement appears to work.

Where the saving comes from and where it goes

The reason routing has a reputation for disappointing is that it is usually evaluated on the left column alone, where the arithmetic is genuinely compelling: a large majority of requests, a wide rate spread, an obvious multiplication. The right column is where the difference between volumes shows up, and it is mostly fixed cost. A classifier call is paid on every request whether or not it changed anything, the engineering has to be revisited each time a provider retires or reprices a model, and the sampling is ongoing attention rather than a one-off. Fixed costs against a variable saving means there is a volume below which this is a bad trade, and organisations that adopt routing while still piloting tend to find it there.

Seen in the wild

Choosing between many providers and models behind a single interface, which is what makes routing practical to implement at all.
OpenRouter
An automation using a cheap model to classify incoming records and a capable one only for the ones that need drafting.
n8n
Sending routine local work to a model running on your own hardware and reserving a hosted model for the difficult cases.
Ollama

Common misconceptions

People assume

Routing means always using the cheapest model that works.

In fact

It means matching the model to the request, and for genuinely hard work the capable model is the cheaper choice once rework is counted. A router tuned only to minimise the immediate rate will send difficult cases to something that struggles, which is a saving on the invoice and a cost everywhere else.

People assume

It is a straightforward optimisation.

In fact

It adds a decision that can be wrong, code to maintain, and a second set of behaviour to watch. At high volume that complexity is repaid comfortably. At low volume the classifier calls and the failed attempts frequently exceed what the arrangement saves, and the simpler answer is to pick the right model per workflow by hand.

Telling them apart

Model routing vs AI gateway

Model routing

Deciding which model answers each request.

AI gateway

The single point every request passes through, whatever answers it.

A gateway is where routing usually lives, but it does plenty else and routing can exist without one.

Questions

How is the routing decision actually made?: By rule, by a cheap classifier, or by escalating after a failed attempt. Rules are the most predictable and capture most of the saving; escalation is the most robust where failure is detectable; a classifier sits between and introduces a judgement that can itself be wrong. Most working deployments use rules by task type.
When is it not worth doing?: At low volume, where the classifier calls, the failed cheap attempts and the maintenance exceed the saving. It is also poor value where nearly all the work is genuinely hard, since the cheap path is rarely taken. Choosing the right model per workflow by hand achieves most of it with none of the machinery.
What is the risk?: Quality falling without anybody noticing. A misrouted request returns a worse answer rather than an error, and no one downstream knows which model produced it. Sampling the cheap path's output periodically is the check that keeps the arrangement honest, and it is the first thing dropped once it appears to be working.
Does it help with reliability as well?: It can, since a system that already knows how to send a request elsewhere can send it elsewhere when a provider is unavailable. That is a different mechanism from cost routing and needs deliberate arrangement, but the two share the plumbing and building the first makes the second considerably cheaper.

Key takeaways

Most work is routine and does not need the most capable model.
Rules by task type capture most of the saving with the least risk.
The classifier and the failed attempts come out of the saving.
Misrouting degrades quality silently, so sample the cheap path.

Tools that use this

OpenRouter
Many providers behind one interface, which makes routing practical.
n8n
A cheap model classifying records, a capable one drafting the few that need it.
Ollama
Routine work locally, hosted models reserved for the hard cases.