Glossary
Inference
Inference is running a trained model to produce an answer, as distinct from training it in the first place, and it is the part you pay for every single time somebody uses the thing.
In plain terms
Training is building the factory. Inference is running it. Somebody else paid to build the factory, once, at enormous expense. What you pay for is the power every time something comes off the line, and unlike most software you have bought, that cost does not flatten out the more you use it.
Why it matters
The economics everybody in business has internalised about software, that you build it once and each additional user costs approximately nothing, do not hold here. Every answer costs something, the ten thousandth costs what the first did, and a proposal to run a model across all your historical records is a finance question before it is a technical one. Knowing which half of the word you are paying for makes a great many otherwise confusing conversations about AI budgets straightforward.
How it works
The model's internal settings are fixed by the time you meet it, and this is simply the act of passing your input through them to produce output. Nothing is learnt during it, nothing about the model changes, and the same input with the same settings will produce very nearly the same result tomorrow. It is a calculation rather than an event.
What it costs tracks how much text goes in and how much comes out. That is why a question asked at the end of a long conversation costs more than the identical question asked fresh, and why attaching a large document is a spending decision as much as a convenience. Anything you send is read, and reading is most of the bill on document-heavy work.
It happens somewhere, and choosing where is a genuine decision. On a provider's servers you pay per use and own nothing. On your own hardware you buy capacity up front and then run it as much as you like, which suits steady high volume and suits nothing else, because idle hardware costs the same as busy hardware.
The operation is identical whether it is one person typing in a chat window or a batch running over a hundred thousand documents overnight. The only difference is how many times it happens, which is worth stating because it is why a pilot that felt free can produce an alarming invoice once it is connected to a real queue.
Every optimisation is a way of doing less of it. Reuse repeated material where the provider supports caching it, choose a smaller model wherever it clears your bar, ask for shorter answers, send less context, and do not call a model at all where a rule would do. Those five account for most of the difference between a sustainable system and an expensive one.
Why the usual software arithmetic stops working
Seen in the wild
Send one request through a routing interface where the cost of that request is shown next to the answer, which makes the per-use nature of it immediate.
OpenRouterRun a model on your own machine and notice that the cost has become your hardware and your electricity rather than a line on an invoice.
OllamaPut a model step into an automation that fires on every incoming record, which is where a per-use cost quietly becomes a monthly one.
n8n
Common misconceptions
People assume
The expensive part is training the model.
In fact
For whoever built it, yes, once. For you it is this, permanently. Across the industry the total spent running models has grown to exceed, on some measures considerably, the total spent training them, simply because running happens billions of times and training happens occasionally.
People assume
It will get cheaper for us as we use it more.
In fact
Prices per unit have fallen over time and your own bill still rises with use, because there is no volume at which additional answers stop costing anything. Software buyers expect a plateau and there is not one. Budget it as a variable cost that tracks activity, in the way you would a courier bill.
People assume
Running it on our own hardware makes it free.
In fact
It moves the cost rather than removing it. The hardware is capital, the electricity is real, and idle capacity costs the same as busy capacity. That trade favours steady heavy use and goes badly for occasional use, which is the opposite of what most organisations discover they have.
Telling them apart
Inference vs Pre-training
Inference
Running the finished model. Happens every time anybody asks anything, costs a little each time, and never stops.
Building it in the first place. Happens once, costs an amount few organisations could contemplate, and is somebody else's problem.
Almost every cost conversation you will have is about the first, and almost every headline you read is about the second. That mismatch is why AI budgets are so often misjudged in both directions.
Questions
- Why does our bill keep rising rather than settling?
- Because you are charged for activity rather than for access. Each answer consumes resources and is priced accordingly, so there is no point at which additional use becomes free. That is unlike almost every other piece of software you buy, and it is the defining surprise of a first year of serious use.
- What actually drives the cost?
- How much text goes in, how much comes out, and how capable a model you chose. Long conversations, large attachments, verbose answers and reaching for the largest model by default are the four ordinary habits that multiply a bill, and all four are adjustable without changing what the system does.
- Is it cheaper to run models ourselves?
- Only with steady, high, predictable volume. Owning hardware turns a variable cost into a fixed one, which helps if you are busy all day and hurts if you are not, because idle capacity costs the same. Add the staff time to run it, and the crossover is further away than most spreadsheets assume.
- Does running the model use our data to improve it?
- Not as part of the operation itself, which changes nothing about the model. Whether your inputs are retained and later used for training is a separate question answered by the provider's terms, not by the technology, and it frequently differs between a consumer plan and a business agreement.
Key takeaways
- Inference is running a finished model; nothing is learnt and nothing about the model changes.
- It is charged by activity, so there is no volume at which further answers become free.
- Cost tracks text in and text out, which is why long threads and big attachments are spending decisions.
- Running it yourself converts a variable cost into a fixed one, which only suits steady heavy use.
- Every optimisation amounts to doing less of it, including not calling a model where a rule would do.
Tools that use this
- OpenRouter
Cost shown beside the answer, which makes the per-use nature immediate.
- Ollama
Locally, where the cost becomes your hardware rather than an invoice.
- n8n
A step firing on every record, where per-use quietly becomes per-month.
Last checked July 2026