Could our organisation do this?

Almost certainly not, and almost certainly you should not want to. The computing bill, the time and the specialist staff are beyond nearly everybody, and the result would be worse than what you can rent. The interesting work is in the layers above, which is where a business advantage would actually come from.

Does it explain why the model is bad at our subject?

Usually, yes. If your field is thinly documented in public, or documented mostly in a language the material under-represents, the model read very little about it and cannot know that. Supplying your own material at the time you ask is the remedy, and no amount of later training substitutes for it.

Is our data in there?

Anything of yours that was publicly available might be, and anything internal is not. Whether your conversations with a product become material for a future model of this kind is a separate question answered by that provider's terms, which frequently differ between a consumer plan and a business agreement.

Glossary

Pre-training

Pre-training is the long first stage of building a model, where it learns general patterns from an enormous body of material, before any later stage teaches it to be helpful or to follow instructions.

In plain terms

The reading, before the training. A model at the end of this stage has absorbed a vast amount of material and is useless to talk to, because nothing has yet taught it that a question wants an answer. What it has acquired is grammar, facts as they appeared in the text, styles of argument and the shape of most kinds of document. Everything conversational comes later.

Why it matters

It is where a model's knowledge comes from, which makes it where the cut-off comes from, where the uneven coverage comes from, and where the confidently inherited errors come from. None of those can be changed by anything you do afterwards. Fine-tuning does not fix them, instructions do not fix them, and the remedy for all three is supplying material at the time you ask rather than hoping the model already had it.

How it works

The model is repeatedly shown text and asked to predict what comes next, scored on how close it got, and adjusted. Done at enormous scale, that single exercise produces a working grasp of language and a great deal of incidental knowledge, because predicting the next word well requires knowing something about the subject.

It is the expensive stage by a wide margin, in computing, in time and in specialist staff, which is why relatively few organisations do it and why nearly every product you meet is built on somebody else's. That concentration is an economic fact about the industry rather than a technical one.

The material has an end date and the model has no sense of it. Everything after that point is absent, and the specific failure to watch for is confidence rather than silence: questions about later events are answered in the same assured tone as everything else.

Coverage is inherited exactly as it was found. Subjects the internet documents endlessly are covered in depth; smaller languages, specialist trades and local regulation are thin or absent. The model does not know where its own coverage runs out, which is why it does not warn you.

What each stage put there

Splitting the two stages apart explains a pattern that otherwise looks like a coincidence. Ask why an assistant does not know about last month, why it is thin on your industry, why it repeated an error that has been on the internet for years, and the answer in every case is the left column, which was fixed before anybody sold you anything. Ask why it is polite, why it answers rather than rambling, why it declines some things and hedges others, and the answer is the right column, which was installed afterwards and can be adjusted. That division is why the same remedy keeps appearing across this whole subject. If the problem is knowledge, supply the material with your question, because the stage that would have contained it finished long ago.

Seen in the wild

Ask about something recent and watch whether an assistant says it does not know or answers anyway, which is this stage's end date showing itself.
ChatGPT
Put the same question to a tool that searches first, where the material arrives now rather than from what was read months ago.
Perplexity
Run a published base model that has had no later stage applied and notice how unlike an assistant it is, which is what this stage alone produces.
Ollama

Common misconceptions

People assume

This is where the model learns to be helpful.

In fact

It is where the model learns language and absorbs material. Helpfulness, following instructions and declining certain requests are installed by a later stage, and a model that has only had this one will happily answer a question with three more questions, because that is what a page of questions tends to do.

People assume

A newer model has no cut-off problem.

In fact

It has a later one. Every model has an end date and none of them announces it, so the failure shape is identical and simply moves. Where currency genuinely matters, the answer is a tool that searches and cites rather than a more recent model, which will be stale again by next year.

Telling them apart

Pre-training vs Fine-tuning

Pre-training

The enormous general stage, done once by whoever built the model, at a cost few organisations could contemplate.

Fine-tuning

A small later adjustment on your own examples, affordable, and shaping behaviour rather than knowledge.

One is the reading and the other is the coaching. Almost nobody reading this will do the first, and the second is available to any organisation with a few hundred consistent examples.

Questions

Could our organisation do this?: Almost certainly not, and almost certainly you should not want to. The computing bill, the time and the specialist staff are beyond nearly everybody, and the result would be worse than what you can rent. The interesting work is in the layers above, which is where a business advantage would actually come from.
Does it explain why the model is bad at our subject?: Usually, yes. If your field is thinly documented in public, or documented mostly in a language the material under-represents, the model read very little about it and cannot know that. Supplying your own material at the time you ask is the remedy, and no amount of later training substitutes for it.
Is our data in there?: Anything of yours that was publicly available might be, and anything internal is not. Whether your conversations with a product become material for a future model of this kind is a separate question answered by that provider's terms, which frequently differ between a consumer plan and a business agreement.

Key takeaways

It is the long first stage, where a model absorbs language and material and nothing conversational.
It is the expensive stage, which is why nearly every product is built on somebody else's.
The cut-off, the uneven coverage and the inherited errors all originate here.
None of them can be fixed by later training or by instructions.
The remedy for all three is supplying material at the moment you ask.

Tools that use this

ChatGPT
The end date, visible by asking about something recent.
Perplexity
Material arriving now rather than from what was read.
Ollama
A base model with no later stage, which is unlike an assistant.

Last checked July 2026

All glossary terms