What we'll cover
We have the pieces: pretraining (5.1) and supervised fine-tuning (5.2). This sub-session assembles the whole pipeline, names what each stage contributes to the model you talk to, and, connecting back to Session 3, pinpoints where each stage opens a door to misalignment. Capabilities come from pretraining; the "assistant" is a thin, post-trained layer on top; and every layer is an imperfect proxy for what we meant.
Mandatory readings
• Ouyang et al. (2022), InstructGPT (arXiv:2203.02155): §3 and Figure 2, the three-stage pipeline diagram. You read §3.1 to §3.4 in 5.2; re-read the section opening and the figure now that the whole stack is in view. ≈600 words.
Total mandatory load: ≈600 words.
Optional readings
• Lambert et al. (2024), "Tülu 3: Pushing Frontiers in Open Language Model Post-Training" (arXiv:2411.15124): the open post-training recipe that named reinforcement learning with verifiable rewards.
• Guan et al. (2024), "Deliberative Alignment: Reasoning Enables Safer Language Models" (arXiv:2412.16339): training models to reason over a written safety specification; the specification itself is Session 7.5.
• Bai et al. (2022), "Training a Helpful and Harmless Assistant with RLHF" (arXiv:2204.05862): the helpful/harmless framing of the post-training stages, read fully in Session 6.
Core material
The reference text for this sub-session and all of Session 6 is Nathan Lambert's open textbook Reinforcement Learning from Human Feedback (rlhfbook.com): chapter-length treatments of instruction tuning, reward modelling, policy optimisation and reinforcement learning with verifiable rewards, updated as the field moves. Read the chapters alongside the sub-sessions that name each stage.
The pipeline, end to end
A modern chat model is the product of a sequence of stages, each operating on the output of the last (Ouyang et al., 2022).
Five stages
| Stage | Objective | What it contributes |
|---|---|---|
| 1. Pretraining | Next-token prediction on web-scale text (self-supervised). | Knowledge, language, reasoning: raw capability. Produces the base model. |
| 2. Supervised fine-tuning | Next-token loss on curated (instruction, response) demonstrations. | Instruction-following, the assistant register, format, and a first pass at refusals. |
| 3. Preference optimisation | Optimise a learned model of human preference (RLHF, covered in Session 6, or DPO). | Helpfulness, harmlessness, tone; pushing quality beyond what demonstrations alone reach. |
| 4. Verifiable-reward RL | Reinforcement learning against a programmatic check: a unit test passing, an exact-match answer (RLVR; below). | Long-form reasoning in checkable domains (maths, code); the stage behind "reasoning" models. |
| 5. Deployment wrapping | System prompt, tools, filters, sampling settings. | The persona and constraints at use-time, adjustable without retraining. |
The compute is lopsided, though less starkly than in the 2022-era pipeline: stage 1 is months of training on enormous clusters, and stages 2, 3 and 5 are comparatively tiny. Stage 4 broke the pattern: since the reasoning models of 2024–25, RL on verifiable rewards takes a substantial and growing share of frontier training compute. Even so, almost all the "intelligence" is laid down in stage 1, and almost all the "alignment" is in the small layers on top.
Where each behaviour is shaped
Capabilities → pretraining
What the model can do is fixed mostly in stage 1. Post-training rarely adds new abilities; it surfaces, suppresses, or reshapes what is already latent.
Instruction-following → SFT
The habit of treating the prompt as a request, the answer format, and the default "voice" are installed in stage 2.
Helpful/harmless/honest → preferences
The fine-grained "be more like this" (politeness, caution, refusing well, picking the better of two answers) comes from stage 3 (Session 6).
Reasoning → verifiable rewards
The long chains of working on maths and code problems are trained in stage 4, where the reward comes from a program rather than a rater.
Final persona → the system prompt
"You are a helpful assistant…", tool access and content filters are stage 5: a thin, swappable wrapper, and often the easiest layer for a user to argue around.
The verifiable-rewards stage
For some tasks, correctness can be checked by a program. There, post-training can skip the human judge entirely.
Reinforcement learning with verifiable rewards (RLVR)
Preference optimisation (stage 3) learns a reward model from human comparisons because, for most of what we want ("be helpful", "explain this well"), there is no formula to check an answer against. But a family of tasks does have one: a maths problem with a known final answer, code against a unit-test suite, a formal proof that either compiles or does not. For these, reinforcement learning can use the check itself as the reward: sample an attempt, run the verifier, reward success. Reinforcement learning here means only this: the model produces something, a number scores it, and training makes higher-scoring behaviour more likely. Session 6.1 gives the full account and 6.3 the algorithms, so take the shape on trust for now. Tülu 3 (Lambert et al., 2024) named this recipe reinforcement learning with verifiable rewards (RLVR), and it is the stage behind the "reasoning" models that followed: DeepSeek-R1 (in the readings) learned to produce long chains of working almost entirely through RL against checkable maths and code answers.
The verifier changes the economics of the stage. A human comparison costs an annotator-minute, and the reward model trained on it inherits every labeller bias (Session 6); a unit test costs nothing to run a million times and is exactly as consistent on the millionth run as on the first. Reward signal stops being the scarce input, which is why this stage, alone in the post-training stack, has grown to absorb serious compute.
Verifier hacking
A verifier is still a proxy. A unit-test suite covers the cases its author thought of, so a policy optimised against it learns to pass those cases (special-casing the visible tests, exploiting the checking harness) rather than to be correct in general; an exact-match checker rewards the right number reached by wrong working. Session 3.1's lesson transfers unchanged: replace the learned reward model with a programmatic one and reward-model hacking becomes verifier hacking, harder where the check is tight (a full proof checker), routine where it is loose (a thin test suite).
Where the stage stops
The stage exists only where a check does. Helpfulness, honesty in open conversation, a good summary for a patient: none of these has a verifier, so RLVR runs beside preference optimisation rather than replacing it, each shaping a different slice of behaviour. How much it adds even where it does apply is contested, and 6.3 takes that argument up once you have the algorithm in hand.
Where misalignment enters
Re-read the pipeline through Session 3's lens and each stage is a place the gap between objective and intent can open.
One failure mode per layer
- Pretraining imitates everything. The base objective is "model the data", which includes deception, manipulation, dangerous know-how and every harmful persona on the web. Those capabilities exist in the model before any safety work; alignment must suppress them, not avoid creating them.
- SFT inherits its demonstrators' limits. The model can be no better, and no safer, than the demonstrations, and their blind spots (e.g. almost entirely English-language, English-context refusals) become the model's blind spots.
- Preference optimisation is a proxy. Stage 3 maximises a learned reward model of human approval, not the truth or the user's real interest. That is the specification problem of Session 3.1, and we will see it bite as sycophancy and reward hacking in Session 6.
- Verifiable rewards invite verifier hacking. Stage 4's programmatic checks are proxies too: a policy can special-case the test suite or game the answer-matcher instead of becoming correct (above). The same Goodhart failure as stage 3's, relocated to a different proxy.
- The wrapper is shallow. A system prompt is not a guarantee; it is text the model is trained to weight heavily but can be induced to ignore (jailbreaks, Session 9). Safety that lives only in stage 5 is the most brittle kind.
Alignment is a thin layer over a vast base
A frontier assistant is an enormous, capability-rich, value-indifferent base model, with a comparatively thin layer of fine-tuning shaping how it presents itself. That layer is real and does a great deal of work, but it is small, it is a proxy, and it is applied to a system that already contains the behaviours we are trying to prevent. Much of the rest of this course is about how fragile that layer is (robustness, Week 5), whether we can see through it (interpretability, Weeks 7–8), and whether the underlying model is merely behaving aligned (deception, Session 3.4).
Base vs aligned models
The alignment layer can be removed
Because alignment is a thin fine-tune, it can be largely undone: a few hundred fine-tuning steps can strip the safety behaviours off an open-weights model and recover much of the base model's unfiltered capability. This is central to the open-weights debate (Sessions 10.2 and 10.4): releasing weights releases the capability, and the safety layer that ships with it is not a durable lock. It also sharpens the Global-South thread: the languages and contexts least represented in the (English-centric) alignment layer are where it is thinnest to begin with.
A worked trace: one request through the stack
Follow a single prompt ("Summarise this medical study for a patient") and watch each stage contribute, and each open its own risk.
Where each property (and each hazard) comes from
Pretraining supplies the raw ability: the model has read enough medicine and plain-language writing to be able to summarise, and latently enough to produce confident-sounding nonsense, because the web contains both. SFT makes it attempt the task in the assistant register rather than, say, continuing with more study abstracts, but it can only imitate the summaries its demonstrators wrote, English-centric and of bounded quality. Preference optimisation (Session 6) then pushes toward summaries humans rate highly, which is where it can quietly learn that reassuring, fluent, confident summaries score better than accurate-but-hedged ones (sycophancy). The verifiable-rewards stage barely features: no program can check a patient summary, so this request sits in the slice of behaviour that only human preference ever shaped. Finally the system prompt ("you are a careful medical assistant; add a disclaimer") shapes the surface, and is the layer a user can try to argue away ("ignore previous instructions…").
The helpful behaviour you see is assembled from four sources, and the failure you fear (a confident, wrong, un-hedged summary that a patient trusts) can originate at any one of them. Asking "which stage would this failure come from?" is a useful diagnostic, and one this course returns to.
Locating a failure in the pipeline
Different stages call for different fixes. A capability that shouldn't exist at all is a pretraining/data problem (or an unlearning one, Session 10). A model that won't follow instructions is an SFT problem. Sycophancy and reward hacking are preference-optimisation problems (Session 6). A jailbreak that peels off the persona is a robustness problem (Session 9). Locating a failure in the pipeline tells you which tool can address it, and warns you when a fix at one layer (a sterner system prompt) is being asked to paper over a problem rooted in another (a capability baked in at pretraining).
Questions to bring to class
- Pick a real assistant failure you've seen. Which stage of the stack do you think it came from, and what evidence would confirm it?
- "Almost all the intelligence is laid down in stage 1; almost all the alignment is in the small layers on top." What does this predict about how hard it is to make an open-weights model permanently safe?
- The same harmful capability is present in the base model and suppressed by the alignment layer. List the distinct ways that suppression could fail.
- If you could only strengthen one stage for safety, which would you choose, and what would you be giving up?
Next
In the 5.4 lab you'll find each of these stages in the source of Karpathy's nanochat, then put a base model beside its instruction-tuned sibling and work out what the tuning changed. The answer is less tidy than the completes-versus-complies story suggests.