How much should we reserve?

The floor rather than the peak: the volume that genuinely runs all the time. Peaks can overflow onto per-request capacity, which gives you predictability where the load is constant without paying the reserved rate through every quiet hour of the week.

What does it actually improve?

Consistency, not speed. Answers do not arrive any faster, but the time they take stops depending on how busy the shared capacity is, which matters wherever a person is waiting for a reply or a downstream step has a timeout it can trip.

What is the hidden cost?

Being tied to what you reserved. Capacity is usually bought against a specific model, so switching to something newer or cheaper turns from a configuration change into a commercial conversation, which is an awkward constraint to hold while models keep improving quickly.

Glossary

Provisioned throughput

Paying to reserve a guaranteed amount of model capacity, charged whether or not the work turns up to use it.

In plain terms

Booking capacity in advance so it is there when you need it. You get predictable performance and you pay the same whether the work arrives or not, which makes it a bet on how busy you are going to be.

Why it matters

Because it is the point at which the arrangement stops flexing to fit you and starts requiring you to fit it. That is a real change in how a team operates, and it is usually presented as a pricing decision rather than the operational commitment it is.

How it works

What you are buying is predictability rather than speed. A reservation does not make an individual answer arrive faster; it makes the arrival time stop depending on how busy everybody else is, which is a different and often more valuable property.

You pay for the reservation, not for the use. Quiet hours cost exactly what busy ones do, so the calculation is about your average against the reserved level rather than about your peak against it.

That makes it a bet on your own demand curve, and the usual way it is lost is by reserving for the peak. Sizing to the busiest hour means paying that rate through every quiet one, and demand is almost always lumpier than the people running it assume.

The workable pattern is to reserve the floor and let the rest overflow. Cover the volume that genuinely runs all the time, and take the peaks on per-request capacity, which gives predictability where it matters without buying silence.

It ties you to what you reserved. Capacity is generally bought against a particular model, so moving to a newer or cheaper one becomes a commercial decision rather than a configuration change, and that constraint arrives at the worst time in a fast-moving field.

A reservation you have paid for changes how a team behaves, and not always well. Capacity that is already bought looks free at the point of use, so work gets pushed onto it that nobody would have approved at per-request prices, and the reservation is then found to be too small.

The trigger to consider it is evidence rather than volume. A specific place in the product where variable response times cause a real problem, or a steady load large enough that the arithmetic favours reserving, are the two honest reasons.

Two ways to size a reservation

The reason the left-hand column wins so often is that the decision is usually made just after an incident, when somebody has watched a queue back up and wants it not to happen again. That is exactly the wrong moment to size a reservation, because the peak is vivid and the ninety quiet hours around it are not. A month of ordinary data answers the question properly: plot the load by hour, find the level below which it essentially never drops, and reserve that. Everything above it is by definition intermittent, and intermittent load is precisely what per-request capacity is good at. The second consideration is worth weighing at the same time rather than later, because it is the one that bites months afterwards. A reservation is generally tied to a specific model, and in a field where a better or cheaper option appears regularly, a commitment that turns switching into a commercial negotiation has a cost that does not appear anywhere on the invoice.

Seen in the wild

Serving a model yourself, where the capacity is unambiguously something you pay for.
vLLM
Keeping per-request access for the peaks while a floor of work runs on reserved capacity.
OpenRouter
A customer-facing feature where response times varying by the minute is the actual problem.
Sierra

Common misconceptions

People assume

It makes things faster.

In fact

It makes them predictable. An individual answer is not produced more quickly; what changes is that the time it takes stops depending on how much load everybody else is putting on the same shared capacity.

People assume

Reserve for the peak and you are covered.

In fact

You are also paying that rate through every quiet hour, and demand is lumpier than most teams believe. Reserving the floor and overflowing the peaks is almost always the better shape of the same decision.

Questions

How much should we reserve?: The floor rather than the peak: the volume that genuinely runs all the time. Peaks can overflow onto per-request capacity, which gives you predictability where the load is constant without paying the reserved rate through every quiet hour of the week.
What does it actually improve?: Consistency, not speed. Answers do not arrive any faster, but the time they take stops depending on how busy the shared capacity is, which matters wherever a person is waiting for a reply or a downstream step has a timeout it can trip.
What is the hidden cost?: Being tied to what you reserved. Capacity is usually bought against a specific model, so switching to something newer or cheaper turns from a configuration change into a commercial conversation, which is an awkward constraint to hold while models keep improving quickly.