Glossary
RLHFreinforcement learning from human feedback
RLHF is a training stage in which people compare a model's competing answers and say which is better, after which the model is trained towards the kind of answer they preferred.
In plain terms
The first stage of training teaches a model to continue text, which produces something knowledgeable and useless: ask it a question and it might write three more questions, because that is what a page of questions looks like. This stage is where people show it, thousands of times over, that when somebody asks a question the preferred continuation is an answer. Almost everything you think of as an assistant's manner was installed here.
Why it matters
It explains the behaviour people complain about, and complain about wrongly. An assistant that agrees with you too readily, hedges when it should commit, apologises constantly, or refuses something entirely harmless is not broken and is not being cautious on its own initiative. It is doing what it was rewarded for. Length and agreeableness were rewarded by raters on average without anybody intending it; refusal behaviour was shaped deliberately, through guidelines that reward caution, and then overgeneralised to requests nobody had in mind. Understanding that these are learnt preferences rather than technical limits changes what you do about them, which is mostly to instruct around them rather than wait for a fix.
How it works
It starts with comparisons rather than corrections. People are shown two or more answers to the same request and asked which is better. Nobody writes the ideal answer, which would be slow and would only ever produce one example; ranking is quick, and a great many rankings describe a preference far more completely than a handful of model answers could.
Those judgements are used to train a second model whose only job is to predict what a person would prefer. That predictor then stands in for human raters at a scale no group of people could reach, scoring the main model's attempts millions of times. The whole approach exists because human attention is the expensive ingredient and this is how you stretch it.
The main model is then adjusted to produce answers that score well, while being held close to what it was before. Both halves matter. Optimising purely for the score produces something that games the rater, and models pushed too hard in this stage become repetitive, evasive or strangely flattering.
Whatever the raters liked becomes the assistant's character, including the things they did not realise they were rewarding. Longer answers tend to read as more thorough, agreement reads as more helpful, and a hedge is rarely marked wrong. Those small preferences, repeated across enormous numbers of comparisons, are where the verbosity, the agreeableness and the over-cautious refusals you notice all come from.
Variants of the idea are now common and the label is applied loosely. Some use written principles instead of case-by-case human judgement, some use another model as the rater, and most production systems combine several approaches. The shared shape is what matters: preferences are collected, turned into a score, and the model is trained towards that score. What reaches you is the result, whoever or whatever supplied the preferences.
The same question, before and after this stage
Seen in the wild
Rate an assistant's answer with the thumbs control in the interface, which is a small contribution to exactly the kind of comparison data this stage runs on.
ChatGPTDisagree with an assistant twice in a row about something you are wrong about, and watch how readily it comes round, which is agreeableness that was rewarded rather than reasoning that failed.
ClaudeRun a published model on your own machine and compare a version tuned for chat with a raw one, where the difference in manner is almost entirely this stage.
Ollama
Common misconceptions
People assume
My thumbs-up and thumbs-down retrain the model straight away.
In fact
Nothing changes in the moment. Feedback is collected, aggregated and used in later training rounds if it is used at all, so the model you are talking to is unaffected by anything you rate. The immediate effect of a rating is on a future version, not on this conversation.
People assume
Refusals and hedging are the model being careful.
In fact
They are learnt behaviour from this stage, and they overshoot in both directions. The same process produces refusals on entirely harmless requests and confident answers where hesitation was warranted. Treating them as a considered judgement about your particular request reads far more intent into them than is there.
People assume
It makes the model more accurate.
In fact
It makes the model more preferred, which is not the same thing and sometimes the opposite. An answer that sounds assured and agrees with the person asking tends to be rated well whether or not it is right, so this stage can reward exactly the manner that makes a wrong answer hard to spot.
Telling them apart
RLHF vs Fine-tuning
RLHF
Trains from comparisons: this answer was better than that one. Shapes broad manner and disposition, and is done by the provider before you meet the model.
Trains from examples: given this input, produce this output. Shapes a specific style or format, and is something you can commission for your own use.
Both change behaviour rather than knowledge. The difference is that one learns from a ranking and one from a worked example, and only the second is normally available to you.
Questions
- Who are the people doing the rating?
- Contracted raters working to written guidelines, in numbers large enough that no individual matters much and small enough that their collective habits show. Providers publish guidelines to varying degrees. It is a reasonable question to ask of a vendor, because those guidelines are effectively the specification for how the assistant behaves.
- Is this why assistants agree with me so easily?
- Largely, yes. Agreement is rated as helpful more often than disagreement, across enormous numbers of comparisons, and the model learns accordingly. The practical countermeasure is instructing explicitly for it, asking for the strongest case against your position rather than for an opinion on it.
- Can we do this ourselves for our own model?
- It is possible and rarely sensible for an ordinary business. It needs comparison data at scale, careful guidelines and expertise in a training procedure that is easy to get wrong. Where a behaviour needs changing, standing instructions and worked examples get most organisations most of the way for a fraction of the effort.
- Does it make the model safer?
- It is one of the main levers for it, and it works partly. This stage is where a great deal of refusal behaviour and tone moderation is installed. It is not a guarantee, it can be worked around, and its failures run both ways, blocking harmless requests as well as occasionally permitting things it should not.
Key takeaways
- It trains a model from comparisons between answers rather than from corrected examples.
- Human rankings train a predictor of preference, which then scores the model at a scale people could not.
- Whatever the raters liked becomes the assistant's character, including what they rewarded unintentionally.
- It optimises for being preferred, which is not the same as being right and can be its opposite.
- Your thumbs-up changes nothing in the moment; it is material for a later training round at most.
Last checked July 2026