Glossary
GPUgraphics processing unit
A GPU is the kind of chip AI runs on, built to perform enormous numbers of simple calculations simultaneously, which happens to be exactly the shape of the work a model does.
In plain terms
A chip designed for drawing pictures that turned out to suit this instead. The ordinary processor in your machine does a few complicated things quickly, one after another. These do thousands of simple things at once, which is what training and running a model consists of, so a component built for video games became the foundation of an industry more or less by accident.
Why it matters
You will never buy one and you will keep meeting the word, in a provider's pricing, in an apology about capacity, and in every argument about whether to run something yourself. Knowing what it is turns those from opaque into ordinary. It also explains a cost structure that otherwise looks arbitrary: this hardware is scarce and expensive, everything you are quoted is priced from it, and the phrase we are capacity constrained is about physical objects rather than about software.
How it works
The work a model does is millions of small multiplications and additions that do not depend on each other, which means they can all happen at the same time. That is precisely what these chips were built for, and it is why they are hundreds of times faster at this than an ordinary processor rather than slightly faster.
How much fast on-chip storage they have is usually the binding constraint rather than raw speed. A model has to fit while it runs, and a chip with plenty of speed and too little room simply cannot host it, which is why the specification you should be watching is the capacity rather than the performance number.
Training and running are different demands. Training needs many of them working together for a long time, which is why it is concentrated in a few organisations. Running a trained model needs far less, which is why it can happen on a laptop, in a phone or beside a camera, and why the two halves of this subject have such different economics.
Supply has been genuinely tight for years, which is unusual for a component and has consequences you feel. It sets the floor under what providers charge, it is why capacity is rationed at busy times, and it is one of the reasons that choosing to run your own means buying into a market where availability is not guaranteed.
Why this chip and not the ordinary one
Seen in the wild
Run a model on your own machine and watch which specification actually decides what will load, which is the on-chip capacity rather than the speed.
LM StudioTry a model too large for your hardware and read the error, which is the clearest possible explanation of the binding constraint.
OllamaCompare what different providers charge for the same model and notice how little the prices differ, which is the shared hardware cost underneath.
OpenRouter
Common misconceptions
People assume
A faster chip is the thing to buy.
In fact
Capacity usually decides what you can run and speed decides how quickly it answers, in that order. A very fast chip that cannot hold your model runs nothing at all, so the first specification to check is how much fits and the second is how quickly it goes.
People assume
We need this hardware to use AI.
In fact
Only to run models yourself. Using a provider's service means their hardware, and that is what almost every organisation does. This becomes your problem when you decide it should, which is a deliberate choice with a purchasing conversation attached rather than a requirement of the field.
Telling them apart
GPU vs Inference
GPU
The hardware: a physical component, in short supply, which everything in this field is ultimately priced from.
The activity: running a trained model to produce an answer, which is what consumes that hardware's time.
One is the meter and the other is the electricity. When a provider raises prices or rations capacity, the explanation is nearly always on this side.
Questions
- Do we need to buy any?
- Only if you have decided to run models on your own hardware, and that decision should be made on cost and data-residency grounds rather than on necessity. Using a provider's service means using theirs, which is what nearly every organisation does and is usually the right answer for anything other than steady heavy load.
- What specification actually matters?
- How much fits, first, because a model that does not fit does not run. Then how fast it goes, which decides your wait. Working out which model you intend to run, and what it needs to be held while running, is the sequence that produces a sensible purchase rather than an impressive one.
- Why is capacity always tight?
- Demand has outrun supply for years, and manufacturing this kind of component is not something anybody can scale quickly. That is why providers ration at busy times, why prices have a floor under them, and why an organisation buying its own is entering a market where availability is genuinely uncertain.
Key takeaways
- It does thousands of simple calculations at once, which is exactly the shape of a model's work.
- On-chip capacity usually decides what you can run; speed decides how long you wait.
- Training needs many of them together; running a trained model needs far less, which is why it fits on a laptop.
- Supply is genuinely tight, which sets the floor under prices and explains capacity rationing.
- You only need to buy any if you have chosen to run models yourself.
Tools that use this
- LM Studio
Locally, where capacity rather than speed decides what loads.
- Ollama
The error from a model too large, which explains the constraint exactly.
- OpenRouter
Prices across providers, with the shared hardware cost visible underneath.
Last checked July 2026