What we'll cover
A base model completes text; we want a model that follows instructions. The first and simplest fix is supervised fine-tuning (SFT): keep training with the same next-token objective, but now on curated examples of the behaviour we want. This sub-session explains how SFT works, how instruction tuning makes it generalise to unseen tasks, and the structural limit that no amount of SFT can overcome, which is what motivates RLHF (Session 6).
Mandatory readings
• Ouyang et al. (2022), "Training language models to follow instructions with human feedback" (InstructGPT; NeurIPS 2022; arXiv:2203.02155): read §3.1 to §3.4, which cover the high-level method, the dataset, the task distribution and how the demonstrations were collected. This is the paper the whole week turns on; you will return to §3.5 in Session 6.2. ≈1,350 words.
Total mandatory load: ≈1,350 words.
Optional readings
• Wei et al. (2022), "Finetuned Language Models Are Zero-Shot Learners" (FLAN; ICLR 2022; arXiv:2109.01652): instruction tuning generalising to tasks never seen in fine-tuning.
• Chung et al. (2022), "Scaling Instruction-Finetuned Language Models" (arXiv:2210.11416): the same idea at scale, across 1,800 tasks.
Supervised fine-tuning, mechanically
SFT is not a new algorithm. It is the same cross-entropy training as pretraining, on different, chosen data.
We assemble a dataset of demonstrations: prompts paired with high-quality target responses, written or curated by people. Then we continue training the base model with the next-token loss, but compute it only over the response tokens \(y\) given the prompt \(x\):
That is the whole mechanism: show the model thousands of examples of "when asked like this, a good answer looks like that", and gradient descent shifts it toward producing such answers. In InstructGPT (Ouyang et al., 2022), this is step 1 of the pipeline (fine-tune the base model on labeler-written demonstrations of the desired behaviour), and it already transforms a raw completion engine into something that broadly does what it's told.
The effect of SFT
The base model already knew how to write a good answer; that capability was latent in the pretraining. SFT mostly teaches it which of its many behaviours to surface by default: respond in the assistant register, follow the instruction, use the expected format. It is less "teaching new skills" than "selecting a persona and a habit of compliance" from the distribution the base model already contains.
Instruction tuning: generalising to unseen tasks
The striking finding is that demonstrations of many tasks teach the model to follow instructions for tasks it never saw.
Wei et al. (2022, "Finetuned Language Models Are Zero-Shot Learners", the FLAN work) fine-tuned a large model on 60+ NLP datasets, each phrased as a natural-language instruction, and found that this instruction tuning substantially improved zero-shot performance on held-out task types, beating zero-shot GPT-3 on a majority of the tasks tested. Chung et al. (2022, "Scaling Instruction-Finetuned Language Models") then scaled the recipe to ~1,800 tasks and added chain-of-thought data, with consistent gains. Instruction-following is itself a skill that generalises once you demonstrate enough of its variety. This is why a modern "instruct" or "chat" model feels qualitatively different from a base model: it has been taught the general habit of treating the prompt as a request to be fulfilled.
The wall SFT hits
SFT can only ever imitate the demonstrations. That is a real ceiling, and the reason the pipeline doesn't stop here.
The limits of demonstration
SFT needs a target to imitate. But for most of what we want from an assistant (be more helpful, more honest, less likely to fabricate, appropriately cautious), the difficulty is not writing one acceptable answer; it is choosing among many plausible answers the one that is better. SFT has no way to express "response A is better than response B": its loss only ever says "make the target more likely". So three things stay out of reach:
- Comparative quality. Humans find it far easier and more reliable to compare two outputs than to author the ideal one from scratch. SFT cannot use comparisons; it can only use single demonstrations.
- The demonstration ceiling. The model can be no better than its demonstration data. If the best human label is mediocre, or if good behaviour is hard to write down but easy to recognise, SFT caps out there.
- Negative signal. SFT teaches what to say, not what not to. Suppressing a behaviour by only ever showing good examples is indirect and leaky.
The bridge to Session 6
This wall is the opening for reinforcement learning from human feedback. RLHF replaces "imitate this answer" with "humans prefer this answer to that one", a signal that is cheaper to collect, can exceed the quality of any single demonstration, and can carry negative as well as positive information. The price, as Session 6 shows, is that we now optimise against a learned model of human preference: a proxy, with all the Goodhart hazards of Session 3.1.
Where safety first enters
SFT is also where the first, partial safety behaviours are installed: demonstration sets include examples of politely refusing harmful requests, hedging on uncertain claims, and declining out-of-scope tasks. This matters for two reasons. It means a model's "refusal reflex" is, at bottom, a fine-tuned habit over a limited set of demonstrated situations, which is why it can fail off-distribution (a harmful request rephrased in isiZulu, or in an unusual format, looks unlike the refusal demonstrations). And it means the values a model expresses trace back to choices about whose demonstrations were collected and what counted as a "good" answer: the "whose values?" question we take up properly in Session 8.
Why "be honest" resists SFT
SFT's ceiling shows most plainly when you try to instil a property that is easy to recognise but hard to demonstrate.
Suppose we want the model to stop confidently making things up (to be calibrated about what it knows). With SFT we would write demonstrations of honest answers. But consider what that requires. For a question the labeller knows, the ideal answer depends on what the model knows, and the labeller can't see that, so they can't reliably demonstrate "admit uncertainty here but answer confidently there". For a question the labeller doesn't know, they can't write the correct answer at all. And a fluent fabrication and a correct answer look identical as demonstrations: both are just plausible text. SFT's loss, "make this target more likely", has no handle on "and was it actually true?".
The general pattern
SFT works well exactly when good behaviour is easy to write down (format, tone, following an explicit instruction). It works poorly when good behaviour is easy to judge but hard to author: honesty, helpfulness on open-ended tasks, "the better of two reasonable answers". That second category is most of what we care about in an assistant, and it is the category where comparisons (A is more honest than B) are available even when demonstrations (here is the perfectly honest answer) are not. The asymmetry ("judging is easier than producing") recurs far beyond RLHF: it is the same intuition behind scalable oversight (Session 8) and behind why verification matters more than generation when you can't trust the output (Session 11).
Collecting better demonstrations
You can, and labs do: demonstration quality is a real lever. But it caps at human-author quality and at what humans can be bothered (and paid) to write, and it still can't express "not that one". RLHF's bet is that comparisons are cheaper to collect, more reliable, and can carry the model past the best single demonstration. Whether that bet pays off is the subject, and the cautionary tale, of Session 6.
Questions to bring to class
- Give a property you'd want in an assistant that is easy to demonstrate, and one that is hard to demonstrate but easy to judge. Why does the distinction predict where SFT succeeds?
- SFT "selects a persona" from the base model rather than teaching new skills. What evidence from the base-vs-instruct comparison (the 5.4 lab) would support or undermine that claim?
- Refusals are installed as fine-tuned habits over demonstrated situations. Predict three kinds of request where that habit would fail to generalise.
- "Judging is easier than producing." Is that always true? Construct a case where it isn't, and say what that would mean for RLHF.
Next
We now have the two ends: a pretrained base model and a supervised fine-tune. Sub-session 5.3 assembles the full post-training stack, shows where each behaviour (and each opportunity for misalignment) is shaped, and distinguishes a base model from the aligned assistant you talk to.