Glossary
Throughput
Throughput is how much work gets through in a given period, which is the measure that matters for bulk processing where no single response is being waited on.
In plain terms
If nobody is sitting waiting for an answer, how quickly any individual answer arrives does not matter. What matters is how many of them get done by morning. Those are different questions with different answers, and improving one frequently makes the other worse, which is why a system tuned for a chat interface is a poor choice for processing ten thousand records.
Why it matters
Because bulk work is where AI spending and AI usefulness tend to concentrate once an organisation is past piloting, and the instinct carried over from interactive use is the wrong one. Teams optimise the wrong number, choose the wrong plan and conclude a job cannot be done in the time available, when what they had was a limit on how many things could run at once rather than on how fast anything was.
How it works
Running things at the same time is what raises throughput, not making each one faster. A job that processes records one after another is limited by the round trip on each, and the same job running many at once finishes in a fraction of the time with every individual response exactly as slow as before. This is the whole of the subject and it is routinely missed.
What limits how many can run at once is usually a plan term rather than anything technical. Concurrency allowances and request rate limits are set by the provider and are frequently the actual constraint, which means the fix is often a commercial conversation rather than an engineering one. Establishing the limits before building around them saves a redesign.
Deferred or batch processing is the mechanism built for this and is often overlooked. Where work is submitted to be done when capacity allows rather than immediately, throughput is high, individual latency is irrelevant and the rate is frequently better. For overnight work with nobody waiting, that trade is entirely free.
Output length is the quiet throughput cost. Generation proceeds piece by piece, so a task that produces long answers occupies capacity for considerably longer than one producing short ones, whatever the input size. Where a bulk job only needs a classification or a short extraction, asking for exactly that rather than an explanation raises throughput substantially.
Ten thousand records, two arrangements
Seen in the wild
An automation processing a queue of records, where running several at once decides whether the job finishes overnight.
n8nA scheduled bulk run over incoming items, where the plan's concurrent operation limit is the real constraint.
MakeRunning a model on your own hardware, where throughput is bounded by the equipment rather than by a plan term.
Ollama
Common misconceptions
People assume
A faster model means more work gets done.
In fact
Only where requests run one at a time, which is the arrangement worth changing first. Concurrency raises throughput by a much larger factor than picking a quicker model does, and once several are running the per-response speed largely stops being the constraint.
People assume
Throughput and responsiveness are the same problem.
In fact
They frequently pull against each other. Batching and queueing raise volume and make any individual answer arrive later, which is right for overnight processing and wrong for anything a person is waiting on. Deciding which situation you are in comes before any tuning.
Telling them apart
Throughput vs Latency
Throughput
How much gets done in an hour. Matters when nobody is waiting.
How long one answer takes. Matters when somebody is.
Ask whether a person is watching. If not, only the first number is real.
Questions
- How do we make a bulk job faster?
- Run more of it at once before doing anything else. Sequential processing is limited by the round trip on each item, and concurrency changes the total by a far larger factor than a faster model would. Once several are running, the limit is usually a plan term rather than the technology.
- What actually limits it?
- Usually the provider's concurrency allowance or request rate limit, which are commercial terms rather than technical ceilings. That makes the constraint worth establishing before building, because the remedy may be a plan change rather than a redesign of the job.
- Does asking for less help?
- Yes, and more than people expect. Generation proceeds piece by piece, so long outputs occupy capacity for longer regardless of the input. A bulk job that needs a category rather than an explanation should ask for the category, which raises throughput without changing anything else.
Key takeaways
- Concurrency raises throughput; a faster model barely does.
- The limit is usually a plan term, so the fix may be commercial.
- Deferred processing suits work nobody is waiting on, often at a better rate.
- Long outputs cost throughput; ask bulk jobs for exactly what is needed.
Last checked July 2026