Session 5.2 — From predictor to instruction-follower (SFT)

What we'll cover

Supervised fine-tuning (SFT) trains a base model on curated examples of the responses we want. We examine the objective, instruction tuning and generalisation, then ask when comparing candidate answers provides a more useful training signal than writing demonstrations.

Mandatory readings

• Ouyang et al. (2022), "Training language models to follow instructions with human feedback" (InstructGPT; NeurIPS 2022; arXiv:2203.02155): read §3.1 to §3.4, which cover the high-level method, the dataset, the task distribution and how the demonstrations were collected. This is the paper the whole week turns on; you will return to §3.5 in Session 6.2. ≈1,350 words.

• OpenAI (2022), "Aligning language models to follow instructions" (openai.com): the same work written for a general reader, with side-by-side GPT-3 and InstructGPT completions on the same prompts. Read it before the paper: it gives you the shape of the method, and the paper sections above then give you the detail. ≈1,850 words.

Total mandatory load: ≈3,200 words.

Optional readings

• Wei et al. (2022), "Finetuned Language Models Are Zero-Shot Learners" (FLAN; ICLR 2022; arXiv:2109.01652): instruction tuning generalising to tasks never seen in fine-tuning.

• Chung et al. (2022), "Scaling Instruction-Finetuned Language Models" (arXiv:2210.11416): the same idea at scale, across 1,800 tasks.

Supervised fine-tuning, mechanically

SFT is not a new algorithm. It is the same cross-entropy training as pretraining, on different, chosen data.

We assemble a dataset of demonstrations: prompts paired with high-quality target responses, written or curated by people. Then we continue training the base model with the next-token loss, but compute it only over the response tokens \(y\) given the prompt \(x\):

\[ L_{\text{SFT}}(\theta) = -\, \mathbb{E}_{(x,y)\sim \mathcal{D}}\left[ \sum_{i} \log p_\theta\!\left(y_i \mid x,\, y_{<i}\right) \right]. \]

That is the whole mechanism: show the model thousands of examples of "when asked like this, a good answer looks like that", and gradient descent shifts it toward producing such answers. In InstructGPT (Ouyang et al., 2022), this is step 1 of the pipeline (fine-tune the base model on labeler-written demonstrations of the desired behaviour), and it already transforms a raw completion engine into something that broadly does what it's told.

The effect of SFT

SFT can make abilities learned during pretraining easier to elicit, and can also teach from new examples. How much it changes task performance, instruction following or style is an empirical question. The base-versus-instruct lab compares these effects on a small set of prompts.

Instruction tuning: generalising to unseen tasks

The striking finding is that demonstrations of many tasks teach the model to follow instructions for tasks it never saw.

Wei et al. (2022, "Finetuned Language Models Are Zero-Shot Learners", the FLAN work) fine-tuned a large model on 60+ NLP datasets, each phrased as a natural-language instruction, and found that this instruction tuning substantially improved zero-shot performance on held-out task types, beating zero-shot GPT-3 on a majority of the tasks tested. Chung et al. (2022, "Scaling Instruction-Finetuned Language Models") then scaled the recipe to ~1,800 tasks and added chain-of-thought data, with consistent gains. Instruction-following is itself a skill that generalises once you demonstrate enough of its variety. This is why a modern "instruct" or "chat" model feels qualitatively different from a base model: it has been taught the general habit of treating the prompt as a request to be fulfilled.

The wall SFT hits

SFT directly rewards the likelihood of supplied target responses. Preference training instead uses judgements about which candidate response is better.

The limits of demonstration

Ordinary SFT needs a target response. If a rater can recognise an improvement but cannot write it, obtaining demonstrations can be costly. Comparisons can select targets for SFT, or train a preference objective directly. These approaches use the available feedback differently:

Comparisons can be used to select better SFT targets. Preference objectives use the winning and losing responses directly, making different use of the same feedback.
Demonstration quality affects learning, but it is not a hard performance ceiling. A model can generalise across examples, filter noise and draw on pretraining. Poor coverage still leaves behaviour inadequately specified. Weak-to-strong generalisation (Session 8.1) studies how stronger models can exceed weak supervision.
The target-likelihood objective does not directly compare a preferred answer with a rejected one. It can suppress competing responses indirectly through normalisation and the examples selected for training.

The bridge to Session 6

This wall is the opening for reinforcement learning from human feedback. RLHF replaces "imitate this answer" with "humans prefer this answer to that one", a signal that is cheaper to collect, can exceed the quality of any single demonstration, and can carry negative as well as positive information. The price, as Session 6 shows, is that we now optimise against a learned model of human preference: a proxy, with all the Goodhart hazards of Session 3.1.

Where safety first enters

SFT is also where the first, partial safety behaviours are installed: demonstration sets include examples of politely refusing harmful requests, hedging on uncertain claims, and declining out-of-scope tasks. This matters for two reasons. It means a model's "refusal reflex" is, at bottom, a fine-tuned habit over a limited set of demonstrated situations, which is why it can fail off-distribution (a harmful request rephrased in isiZulu, or in an unusual format, looks unlike the refusal demonstrations). And it means the values a model expresses trace back to choices about whose demonstrations were collected and what counted as a "good" answer: the "whose values?" question we take up properly in Session 8.

Why "be honest" resists SFT

Consider a property that a rater can recognise more easily than they can demonstrate, such as a well-calibrated answer to an unfamiliar question.

Suppose we want the model to stop confidently making things up (to be calibrated about what it knows). With SFT we would write demonstrations of honest answers. But consider what that requires. For a question the labeller knows, the ideal answer depends on what the model knows, and the labeller can't see that, so they can't reliably demonstrate "admit uncertainty here but answer confidently there". For a question the labeller doesn't know, they can't write the correct answer at all. And a fluent fabrication and a correct answer look identical as demonstrations: both are just plausible text. SFT's loss, "make this target more likely", has no handle on "and was it actually true?".

The general pattern

Demonstrations are useful when suitable target responses can be obtained. Comparisons may be easier on some tasks, including open-ended responses with several acceptable answers. Neither format guarantees truthful labels. Check the feedback against independent evidence and measure the behaviour learned from it.

Collecting better demonstrations

Improving demonstrations can improve SFT. Labels may come from people, checked model outputs or other procedures, and the model may generalise beyond them. Compare the cost and reliability of these options with preference training on the task you want to improve.

Questions to bring to class

Give a property you'd want in an assistant that is easy to demonstrate, and one that is hard to demonstrate but easy to judge. Why does the distinction predict where SFT succeeds?
SFT "selects a persona" from the base model rather than teaching new skills. What evidence from the base-vs-instruct comparison (the 5.4 lab) would support or undermine that claim?
Refusals are installed as fine-tuned habits over demonstrated situations. Predict three kinds of request where that habit would fail to generalise.
"Judging is easier than producing." Is that always true? Construct a case where it isn't, and say what that would mean for RLHF.

We now have the two ends: a pretrained base model and a supervised fine-tune. Sub-session 5.3 assembles the full post-training stack, shows where each behaviour (and each opportunity for misalignment) is shaped, and distinguishes a base model from the aligned assistant you talk to.