Why can nobody fix one specific wrong fact?

Because there is nowhere to fix. What the system knows is spread across a vast number of connection strengths rather than stored in a place, so there is no entry to edit, and changing behaviour means further training or supplying the correct material at the time of asking. That is why retrieval became the practical answer.

Does the brain comparison ever help?

It helps explain where the name came from and almost nothing else. It sets people up to expect comprehension where there is statistics, and fragility where there is remarkable generalisation, so it produces surprise in two opposite directions. Treating the thing as arithmetic at enormous scale is a better predictor.

Do I need to understand this to make decisions?

Not the mechanism, and it is worth knowing that the knowledge has no location. That single fact explains why models cannot simply be corrected, why grounding them in your documents is the standard remedy, and why nobody can tell you exactly why a particular answer came out the way it did.

What is actually in the file you download?

The learnt connection strengths, in enormous quantity, plus enough description of the arrangement to run them. That is why these files are large, why running one needs real memory, and why possessing the file is genuinely possessing the capability rather than a licence to reach somebody else's copy.

Glossary

Neural network

A neural network is a stack of very simple units passing numbers to one another, and the loose comparison to brain cells has done more to confuse expectations than to explain anything.

In plain terms

A great many very simple pieces, each doing something trivial with numbers, arranged so that the output of one becomes the input of the next. The comparison to brain cells is where it got its name and it is doing real damage, because it makes people expect understanding where there is arithmetic and expect fragility where there is enormous statistical power.

Why it matters

Because the analogy sets expectations that the thing then violates in both directions. People assume something brain-like will grasp meaning and be brittle outside its experience, and what they meet is a system that generalises impressively across unfamiliar material while failing on things any person would find trivial. Dropping the comparison makes both behaviours predictable.

How it works

Each unit does something almost embarrassingly simple: it takes numbers coming in, weights them, adds them up and passes a number on. There is no logic, no rule and no representation of a fact inside any individual unit, and everything the system does comes from the arrangement and the strength of the connections rather than from any part being clever.

The layers build from simple to composite, which is why depth matters at all. Early layers respond to small local patterns, later ones to combinations of those, and the useful behaviour emerges several stages up, which is also why nobody can point at where a particular piece of knowledge is kept.

The connection strengths are learnt by repeated correction rather than designed. The system is shown examples, its answers are compared against what was wanted, and the strengths are nudged, millions of times, until the behaviour is right often enough. Nobody chose any individual value and nobody could explain most of them.

The brain comparison is genuinely loose and worth setting aside. It described an inspiration decades ago and does not describe how these systems work now, so treating them as arithmetic performed at enormous scale predicts their behaviour far better, including both the surprising competence and the surprising gaps.

What the metaphor promises, and what you get

It is worth being precise about why this particular metaphor causes so much trouble, because plenty of technical names are loose without doing harm. The difficulty here is that the comparison implies a specific package of properties that tend to travel together in minds and do not travel together in these systems. A thing that understands is expected to know the limits of its understanding, to reason rather than pattern-match, and to be roughly as good at easy things as at hard ones. These systems break all three: they produce work that looks like reasoning, have no representation of their own boundaries, and can write a competent legal summary while miscounting the letters in a word. Somebody who was never told the brain story finds that odd but not contradictory. Somebody who was finds it inexplicable, and reaches for explanations involving deception or intelligence when the honest description is that the metaphor was never accurate and the arithmetic is doing something genuinely unlike thinking.

Seen in the wild

The hub at the centre of open machine learning, where the trained files themselves can be downloaded and inspected rather than only used.
Hugging Face
An open image family whose weights are downloaded and modified locally, which is the trained arrangement made concrete as a file on disk.
Stable Diffusion
A local runner that loads such a file and serves it from your own machine, where size and speed follow the hardware you have.
Ollama

Common misconceptions

People assume

It works like a brain.

In fact

It borrowed a metaphor decades ago and the comparison has been misleading ever since. The units share almost nothing with actual neurons beyond the picture of signals passing along connections, and expecting brain-like behaviour is what produces surprise in both directions at once.

People assume

Somewhere inside, a fact is stored.

In fact

Knowledge is distributed across an enormous number of connection strengths rather than kept anywhere in particular. There is no location for a fact and nothing to edit if one is wrong, which is why correcting a specific error is a research problem rather than an update.

People assume

Understanding the mechanism explains the behaviour.

In fact

The mechanism is simple and the behaviour is not, because everything interesting emerges from scale and arrangement rather than from any component. Being told exactly what a unit does leaves you no better placed to predict what the whole thing will say, which is uncomfortable and true.

Telling them apart

Neural network vs Transformer

Neural network

The general shape: units, layers, connections.

Transformer

One particular arrangement of that shape.

Every transformer is a neural network and most neural networks are not transformers, which is why the narrower word carries the specific claim.

Questions

Why can nobody fix one specific wrong fact?: Because there is nowhere to fix. What the system knows is spread across a vast number of connection strengths rather than stored in a place, so there is no entry to edit, and changing behaviour means further training or supplying the correct material at the time of asking. That is why retrieval became the practical answer.
Does the brain comparison ever help?: It helps explain where the name came from and almost nothing else. It sets people up to expect comprehension where there is statistics, and fragility where there is remarkable generalisation, so it produces surprise in two opposite directions. Treating the thing as arithmetic at enormous scale is a better predictor.
Do I need to understand this to make decisions?: Not the mechanism, and it is worth knowing that the knowledge has no location. That single fact explains why models cannot simply be corrected, why grounding them in your documents is the standard remedy, and why nobody can tell you exactly why a particular answer came out the way it did.
What is actually in the file you download?: The learnt connection strengths, in enormous quantity, plus enough description of the arrangement to run them. That is why these files are large, why running one needs real memory, and why possessing the file is genuinely possessing the capability rather than a licence to reach somebody else's copy.