Glossary
Latency
Latency is the wait between sending a request and getting the answer back, and with these systems it has two distinct parts: how long before anything appears, and how long until it finishes.
In plain terms
Two different waits go by the same word. There is the pause before the first word shows up, which is what makes something feel slow to a person. And there is the time until the last word arrives, which is what matters when a machine is reading the answer rather than a human. Products work hard on the first because people notice it, and anything you have running unattended is quietly paying the second.
Why it matters
It decides which uses are possible at all rather than merely pleasant. A two-second wait is invisible in a chat window and fatal in a phone call, and no amount of answer quality compensates for a conversation that keeps stalling. It also compounds in a way that catches people out: anything taking several steps pays the wait on every one of them, so a process built from five perfectly reasonable calls can feel unusable to you while every part you measured looks fine.
How it works
The two measurements to keep separate are time to the first word and time to the last. Showing the answer as it is produced makes the first very short and changes the second not at all. For a person reading along that is a large improvement; for a step in an automation that has to wait for the whole answer before doing anything, it is worth precisely nothing.
Most of the starting delay comes from how much has to be read before anything can be written. A long conversation, a large attachment or a big set of retrieved documents all have to be taken in first, which is why the same question is quick in a fresh chat and slow in a long thread. The rest of the time is the answer being produced, roughly in proportion to its length.
Model size and deliberation both cost time directly. A larger model is slower per word by construction, and one that works a problem through before answering is generating a great deal of text you never see, all of which happens before the first visible word. That is why a considered answer can take many times longer than a direct one of the same visible length.
The parts you do not control still count. How far your request travels, and how busy the provider is at that moment, both add to the total and vary through the day. Anything with a hard timing requirement needs measuring at the hours you will actually run it, rather than during a quiet afternoon of testing.
In anything multi-step the waits add up. Each tool call is a full round trip, and each step in an agent is another. Five calls at two seconds is ten seconds before anybody sees anything, which is the usual reason a system feels sluggish while every component looks acceptable when you measure it on its own.
How much wait each kind of use can absorb
Seen in the wild
Send an identical request to a large model and a small one through the same interface and compare the waits alongside the answers.
OpenRouterRun a model on your own machine and watch the starting delay grow as the conversation gets longer, which makes the reading-before-writing part visible.
LM StudioPut a model step inside an automation nobody is watching, where only the total time matters and the appearance of progress buys you nothing.
n8n
Common misconceptions
People assume
Streaming the answer makes it faster.
In fact
It makes it feel faster, which is genuinely valuable for anyone reading along, and it changes nothing about when the answer is complete. For an unattended step that needs the whole thing before it can act, streaming is a presentational feature with no effect on the number that matters.
People assume
A faster model will fix it.
In fact
Only if the model is where the time is going. Where the delay is a large attachment being read, three tool calls in sequence, or a busy provider, changing the model moves a small part of a bigger total. Measure which part is slow before buying a remedy for the wrong one.
People assume
It is a technical detail to sort out later.
In fact
It determines which uses are viable. Voice in particular tolerates far less delay than text, and past a certain point people start talking over the reply or assume the call has dropped, regardless of how good the answers are. Finding that out after building is considerably more expensive than measuring one round trip at the start.
Telling them apart
Latency vs Inference cost
Latency
Time. Falls with a smaller model, shorter answers, less material read and fewer round trips.
Money. Falls with exactly the same four things, which is unusual and convenient.
These two mostly move together, so an optimisation for one is usually an optimisation for the other. The exception is deliberation, which buys quality at the expense of both at once.
Questions
- Why is it slow to start on long documents?
- Because everything supplied has to be read before anything can be written, and a long attachment is a lot of reading. The same question in a fresh conversation with nothing attached will start almost immediately. Where only part of a document is relevant, sending only that part is the most direct fix available.
- Does streaming actually help?
- For a person, considerably: watching words appear makes a five-second wait feel like a fraction of that, and it lets somebody start reading or abandon a wrong answer early. For anything automated, not at all, because the next step cannot begin until the last word has arrived either way.
- What can we actually do about it?
- A smaller model, shorter answers, less material sent, fewer round trips, and caching anything repeated where the provider supports it. Those five cover most of it. In multi-step systems, removing one round trip usually beats making every call marginally quicker.
- How fast does a voice system need to be?
- Fast enough that the gap between somebody finishing a sentence and hearing a reply sounds like an ordinary conversational pause. Beyond that people start talking over it or assume the line has dropped, which is why voice products fill the gap with a short phrase while the rest of the answer is prepared.
Key takeaways
- Two waits share the name: time to the first word, and time to the last.
- Streaming shortens the first and never the second, so it buys nothing for unattended work.
- The starting delay is mostly how much had to be read, which is why long threads are slow to begin.
- Waits add up across tool calls and steps, which is why five acceptable parts make an unusable whole.
- Voice tolerates far less delay than text, and answer quality does not compensate for it.
Tools that use this
- OpenRouter
Large against small on the same request, with the wait visible.
- LM Studio
Locally, the starting delay grows with the conversation in front of you.
- n8n
Unattended steps, where only the total time counts.
Last checked July 2026