The word “training” hides three different processes that happen in sequence, cost wildly different amounts, and change wildly different things about the result. Conflating them is the source of most confused conversations about what these models can and cannot do.
The sequence is: learn language from raw text, learn the shape of a helpful answer from demonstrations, then learn which answers people prefer. Roughly 99% of the compute goes into the first stage. Almost everything a user perceives as the model’s personality comes from the last two.
The clearest way to hold it: pretraining creates the capability; the later stages decide which slice of it shows up.
What goes into each stage, and what each one changes:
The stages run in order and do different jobs. Pretraining is where the knowledge arrives, and where almost all of the money goes. The two later stages are cheap and small, and they are responsible for nearly everything a user notices.
Stage 1: pretraining, where the knowledge comes from
Take an enormous pile of text — web pages, books, code, papers, forums. Show the model a fragment, ask it to predict the next token, and nudge the weights whenever it is wrong. Repeat trillions of times.
No labels, no human in the loop, no notion of a question or an answer. Just text and prediction error. This is where the model learns grammar, facts, arithmetic, the syntax of six programming languages, how a legal contract is structured, and how an argument on a forum tends to go. It is also where essentially all of the cost lives: months of compute across thousands of accelerators.
What comes out is called a base model, and it is worth understanding exactly what it is, because it is not an assistant. Ask a base model “What is the capital of France?” and a perfectly reasonable output is:
What is the capital of France?What is the largest city in Germany?What is the currency of Japan?It has continued the text. Somewhere in its training data there was a list of quiz questions, and a list is a very likely continuation of a question. The base model is not being unhelpful; it has no concept of helpful. Its capability is complete at this point — it knows the medicine, the code, and the history. It just has no idea that it is supposed to be someone answering you.
Stage 2: supervised fine-tuning, learning the format of an answer
Now hire people to write conversations. A prompt, and the response a good assistant should give. Tens of thousands of them, covering questions, refusals, code, edits, admissions of uncertainty.
Then keep training with the identical next-token objective — only now the text is exclusively these demonstrations. Nothing about the mechanism changes. The data does.
The effect is dramatic and easy to misread. The model starts answering questions, adopting a consistent voice, and declining some requests. It looks like it learned to be helpful. What it learned is that the likeliest continuation of a question, in this text distribution, is an answer rather than another question.
Two things follow, and they explain a lot.
No meaningful new knowledge arrives here. The data is tiny compared to pretraining. If a fact was not in the pretraining corpus, fine-tuning on a few thousand conversations will not put it there. Fine-tuning teaches behaviour, not content — which is why fine-tuning is the wrong tool for “make the model know our internal docs” and retrieval is the right one.
The assistant is a character the model plays. The weights still model all of human text, including the parts that are hostile, wrong, or unhinged. Fine-tuning makes the helpful-assistant continuation overwhelmingly likely. That framing predicts jailbreaks better than any other: a prompt that makes some other character the likelier continuation gets some other character.
Stage 3: preference tuning, learning what people prefer
Demonstrations have a ceiling. It is hard to write the perfect answer to “explain quantum entanglement to a curious teenager”, and easy to look at two answers and say which is better. Stage three uses that asymmetry.
The pipeline, in its usual form:
- Sample several responses from the fine-tuned model for the same prompt.
- Have humans rank them.
- Train a separate reward model to predict those rankings — a model whose output is a score for how much a human would like a response.
- Optimise the language model to score well under that reward model, with a penalty for drifting too far from where it started.
This is the RL part, and where the name RLHF — reinforcement learning from human feedback — comes from. The recipe predates chatbots: Christiano and colleagues used it in Deep Reinforcement Learning from Human Preferences in 2017 to train simulated robots from nothing but people picking the better of two clips. Newer variants skip the separate reward model and optimise on the preference pairs directly, which matters to people training models and barely at all to people using them.
What this stage changes is taste. Length, hedging, formatting, tone, how readily the model refuses, how often it says “it depends”. These are all things that are hard to specify and easy to judge, which is exactly what preference data is good at capturing.
It is also where the well-known pathologies come from. Optimising against a learned proxy for human approval is a textbook case of a metric becoming a target. Raters mildly prefer longer answers, so answers get long. Raters prefer agreeable answers, so sycophancy shows up — not as a personality flaw but as a scoring artefact. Raters cannot check every claim, so confident wrong answers can outscore honest uncertainty. It is the same failure as optimising the wrong metric, with humans as the label source.
Why the distinction is worth carrying around
| Stage | Data | Cost | What it changes |
|---|---|---|---|
| Pretraining | Trillions of tokens of raw text | Months, enormous | Knowledge and raw capability |
| Supervised fine-tuning | ~10⁴–10⁵ written conversations | Days | Behaviour: answering, format, refusals |
| Preference tuning | Human rankings of model outputs | Days to weeks | Taste: tone, length, agreeableness |
Most practical questions resolve against this table. Can fine-tuning teach the model our product? No — that is knowledge, and it belongs in the context window. Why is it so agreeable? Stage three. Why does it invent citations? Stage one never distinguished true from likely, and no later stage installed the distinction. Why does a jailbreak work? The assistant is a costume over a model of all text.
The deepest walk-through of the full pipeline is Deep Dive into LLMs like ChatGPT by Andrej Karpathy — long, unhurried, and worth the time for anyone who wants the details behind this sketch.
The takeaway that survives the next model release: what these systems know was fixed months before anyone chose their personality, and no amount of work on the personality changes what they know.