Why can it handle a reference from much earlier in a document?

Because every part of the input is available to every part of the output rather than having to survive being passed along in order. Distance across the input stops being the limiting factor, which is precisely what earlier approaches struggled with.

Why are long inputs so expensive?

Because everything is weighed against everything else, and that grows faster than the length itself does. Doubling the input more than doubles the work, which is why sending one very long document costs disproportionately more than sending several short ones.

Do we need to understand this to use these tools?

No, and it makes a surprising amount of otherwise arbitrary behaviour legible. Why length costs what it does, why position in a document matters less than people expect, and why long-context pricing is shaped the way it is all follow from this one idea.

Glossary

Attention mechanism

The mechanism that lets a model weigh which parts of an input matter for each piece of output it produces, whatever their distance apart.

In plain terms

A way for the system to look at everything it was given and decide, for each word it writes, which parts of the input matter most. The important part is everything: a word near the start is as reachable as one from a moment ago.

Why it matters

Because it explains two things people notice separately and never connect: why these systems handle a reference to something twenty paragraphs earlier, and why sending a very long input costs so much more than sending a short one.

How it works

Every part of the input can be weighed against every other part. Rather than reading in order and carrying a summary forward, the mechanism considers the whole input at once and decides what is relevant to each piece of output.

That is why distance stopped being decisive. Earlier approaches degraded as the relevant material got further away, because information had to survive being passed along; here a detail at the beginning is as reachable as one at the end.

The weighing is what the word attention names. For each thing being produced, some parts of the input count more than others, and which parts count is learned rather than specified, which is why nobody wrote a rule about pronouns.

The same property is the source of the cost. Considering everything against everything grows faster than the length itself does, so doubling an input more than doubles the work, which is why long-context use is disproportionately expensive.

It is the idea the transformer design is built around, which is why the term appears in explanations of almost every current model. Knowing it is not necessary to use these tools and does make a great deal of otherwise arbitrary behaviour legible.

It is also why order matters less than people assume but not nothing. Everything is available, and where a thing sits is still information the mechanism can use, so material placed prominently is not equivalent to the same material buried, even though both are reachable.

The weights are not an explanation of the answer. It is tempting to read what a model attended to as its reasoning, and the honest position is that it describes where the mechanism looked rather than why the output was what it was.

Two ways to handle a long input

The connection between the two is the useful part of understanding this at all. A buyer notices that an assistant handles a document well enough to answer a question about something mentioned on page one, and separately notices that the bill for long documents is worse than proportional, and files those as a pleasant feature and an unpleasant pricing decision. They are the same property seen from two sides. That reframing has a practical consequence: the way to reduce the cost of long inputs is to send less rather than to negotiate, because the expense is structural rather than commercial. Retrieving the few relevant passages and sending those, instead of sending everything and relying on the mechanism to find them, is cheaper for a reason that has nothing to do with any supplier's price list and will remain true whatever that list says next year.

Seen in the wild

An assistant resolving a reference to something mentioned much earlier in a document.
ChatGPT
A long input costing disproportionately more than a short one.
OpenRouter
Model documentation describing the design in these terms.
Hugging Face

Common misconceptions

People assume

The model reads left to right and remembers.

In fact

It considers the whole input together and weighs which parts matter for each piece of output. That is why something from the first paragraph is as available as something from the last, which sequential reading could never manage well.

People assume

What it attended to explains the answer.

In fact

It shows where the mechanism looked, not why the output came out as it did. Treating those weights as reasoning attributes an explanation to something that was never producing one.

Questions

Why can it handle a reference from much earlier in a document?: Because every part of the input is available to every part of the output rather than having to survive being passed along in order. Distance across the input stops being the limiting factor, which is precisely what earlier approaches struggled with.
Why are long inputs so expensive?: Because everything is weighed against everything else, and that grows faster than the length itself does. Doubling the input more than doubles the work, which is why sending one very long document costs disproportionately more than sending several short ones.
Do we need to understand this to use these tools?: No, and it makes a surprising amount of otherwise arbitrary behaviour legible. Why length costs what it does, why position in a document matters less than people expect, and why long-context pricing is shaped the way it is all follow from this one idea.