Does it reduce cost or latency?

Neither. The same work is done, the same amount is charged, and the final word arrives when it would have anyway. What it changes is what the person experiences during the wait, which makes it an interface decision rather than a performance one.

When should we not use it?

Wherever the answer is meant to be checked before anybody sees it. Filtering, verification and correction all have to happen before the first word appears, so streaming and checking are alternatives rather than things you can have together on the same surface.

What happens if it fails halfway?

Whatever you designed, and most designs have not considered it. A response that stops partway leaves text on screen that reads as finished, so an interface needs a visible way of saying that what is showing is incomplete rather than merely short.

Glossary

Streaming

Returning an answer piece by piece as it is produced rather than waiting for the whole thing, which changes how the wait feels.

In plain terms

Showing the answer as it is written instead of waiting for the end. It arrives at exactly the same time either way. What changes is that somebody is reading during the wait instead of watching a spinner, which is a large difference in experience and none at all in speed.

Why it matters

Because it is the cheapest improvement available to most interfaces, and because it quietly forecloses a safety option. Once the first words are on screen you have committed to the answer, and any checking you meant to do had to happen before that moment.

How it works

The total time is unchanged. The last word arrives when it would have arrived anyway, and what has changed is that the reader was occupied for most of the wait instead of staring at nothing.

The effect is large because waiting is subjective. A pause with nothing happening feels considerably longer than the same pause spent reading, so the perceived improvement is real even though the measured duration is identical.

It commits you to the answer. Text already on screen cannot be withdrawn without the reader noticing, so any check on the output has to happen before the first word appears, which streaming has just made impossible.

That trade should be made deliberately rather than by default. Where an answer is going to be checked, filtered or corrected before a person acts on it, showing it as it forms undoes the point of the check.

Failures mid-answer are the awkward case. A response that stops halfway leaves a partial answer on screen that reads as complete, and an interface without a defined behaviour for that will show somebody half a sentence and let them act on it.

It changes what a person does with the wait, which has a second effect nobody plans. Readers begin acting on the first sentence before the last one arrives, so an answer whose important qualification comes at the end is frequently read without it, and that is a writing problem rather than a technical one.

It affects nothing about cost. The same work is done and the same amount is charged, so this is purely an interface decision, which is why it is worth making on interface grounds rather than as a performance measure.

The choice streaming actually presents

The second line is the one that costs something, and the cost is invisible until the day it matters. A team that later decides to filter answers, or to check a claim against a system of record, or to suppress a certain kind of response, finds that the architecture already showed the reader everything before any of that could run. Retrofitting a check then means removing the streaming, which is a visible downgrade that users notice and complain about, so it usually does not happen and the check is quietly dropped instead. That is a decision about safety being made by an interface choice taken months earlier for reasons that had nothing to do with it. The recommendation is not to avoid streaming, which would be silly, but to be honest at the outset about which category the surface is in. Anything customer-facing where a wrong answer has consequences deserves the question asked deliberately; an internal assistant where the reader is the check does not.

Seen in the wild

An assistant reply appearing as it is written, so the wait is spent reading.
ChatGPT
A support widget where showing an answer before checking it is the wrong trade.
Tidio Lyro
Code appearing as it is produced, where partial output is still useful to read.
Claude Code

Common misconceptions

People assume

It makes the model faster.

In fact

The last word arrives at the same moment either way. What changes is that the reader spends the wait reading rather than waiting, which is a genuine improvement in experience and no change at all in speed.

People assume

It is always the better choice.

In fact

It removes the chance to check an answer before anybody sees it. Anywhere the output is meant to be filtered, corrected or verified first, showing it as it forms defeats the purpose of doing so.

Questions

Does it reduce cost or latency?: Neither. The same work is done, the same amount is charged, and the final word arrives when it would have anyway. What it changes is what the person experiences during the wait, which makes it an interface decision rather than a performance one.
When should we not use it?: Wherever the answer is meant to be checked before anybody sees it. Filtering, verification and correction all have to happen before the first word appears, so streaming and checking are alternatives rather than things you can have together on the same surface.
What happens if it fails halfway?: Whatever you designed, and most designs have not considered it. A response that stops partway leaves text on screen that reads as finished, so an interface needs a visible way of saying that what is showing is incomplete rather than merely short.