When should we move away from it?

When volume is steady enough to forecast and large enough that per-request pricing exceeds reserved capacity, or when variable response times are causing a problem you cannot design around. Both are specific, and neither is a feeling that something more serious is needed.

Why did it get slower without anything changing?

Because capacity is shared and you hold no reservation. Your busy period frequently coincides with everybody else's, so behaviour observed in quiet conditions is not a guide to what you will see at the moment it matters most, and nothing on your side has changed to explain it.

Does it need a spending limit?

More than most arrangements, because there is no natural ceiling. The same property that makes a quiet month free makes a runaway loop unbounded, and a hard limit set in advance is what converts that into a known worst case.

Glossary

Serverless inference

Model access with no machines to manage, charged for what you send rather than for capacity you have reserved.

In plain terms

Sending work to a model without running anything yourself. You pay for what you use, somebody else worries about the machines, and you have no claim on capacity when everybody wants it at once.

Why it matters

Because it is how almost every organisation starts and how most should stay. The reasons to move away from it are specific and worth being able to name, so that the decision to leave is made on evidence rather than on a feeling that something more serious is required.

How it works

What you are buying is the absence of capacity management. No machines to size, patch, scale or pay for while idle, which removes an entire category of work rather than making it cheaper.

Cost follows use, which is the appeal and the exposure. There is no bill for a quiet month, and there is also no ceiling in a busy one, so it pairs naturally with a hard limit set in advance.

You hold no reservation, which is the part that surprises people. Capacity is shared, and the moment your load peaks is often the moment everybody else's does, so the behaviour you tested in quiet conditions is not the behaviour you get.

The economics invert as volume grows. Paying per request is cheapest while volume is uncertain and becomes the expensive option once it is steady and large, and the crossing point is a calculation rather than a matter of taste.

Latency varies in ways you do not control. There is nothing to tune, so an application sensitive to consistent response times has to absorb the variation in its design rather than engineer it away.

Switching between models costs almost nothing, which is an underrated part of what you are buying. Where capacity is not tied to a particular model, moving to a newer or cheaper one is a configuration change rather than a migration, and that optionality is worth real money while the field is still moving quickly.

Failures and retries cost the same as successes, which shapes how automation should be written. A workflow that retries three times on a hard case pays four times for one answer, so the sensible design gives up early and asks somebody rather than trying harder.

It is the right default for anything whose volume you cannot predict. Uncertain demand is precisely the case where paying for reserved capacity means paying for silence, and most workloads are more uncertain than the people running them believe.

What you gain and what you give up

The mistake worth avoiding is treating the right-hand column as a list of deficiencies to be fixed as an organisation matures. None of those three is a problem until it is: an application that tolerates variable response times is not improved by making them consistent, and reserved capacity bought before volume is predictable is money spent on silence. The useful discipline is to write down, in advance, which of the three would actually trigger a move and what evidence would show it. For price, that is a calculation comparing your steady volume against the cost of reserving. For latency, it is a specific complaint from a specific place in the product rather than a general preference for speed. For peak capacity, it is an incident you can point at. Organisations that skip this step tend to move early, pay for capacity they do not use, and acquire the operational work they were previously paying somebody else to do.

Seen in the wild

Routing requests to whichever model suits, with nothing to run.
OpenRouter
The contrast: serving a model yourself, where capacity is a thing you own.
vLLM
Trying a model on real work before deciding whether it warrants anything more.
Hugging Face

Common misconceptions

People assume

There are no servers.

In fact

There are, and somebody else manages them. What you have bought is the removal of that work from your team, not the removal of machines from the world, and the difference shows in what happens when those machines are busy.

People assume

Paying per request is always cheaper.

In fact

It is cheapest while volume is uncertain, and it becomes the expensive option once volume is steady and large. Where the two cross is an arithmetic question with an answer, not a matter of preference.

Questions

When should we move away from it?: When volume is steady enough to forecast and large enough that per-request pricing exceeds reserved capacity, or when variable response times are causing a problem you cannot design around. Both are specific, and neither is a feeling that something more serious is needed.
Why did it get slower without anything changing?: Because capacity is shared and you hold no reservation. Your busy period frequently coincides with everybody else's, so behaviour observed in quiet conditions is not a guide to what you will see at the moment it matters most, and nothing on your side has changed to explain it.
Does it need a spending limit?: More than most arrangements, because there is no natural ceiling. The same property that makes a quiet month free makes a runaway loop unbounded, and a hard limit set in advance is what converts that into a known worst case.