Session 6.1 — From demonstrations to preferences

Session 6 — RLHF & RL fine-tuning

Session 5 introduced training on demonstrations. Human comparisons provide another signal: a rater may recognise the better of two answers without being able to write either. RLHF learns a reward model from these comparisons and optimises against it, bringing the proxy risks of Session 3 into the training pipeline.

This session's sub-sessions: 6.1 from demonstrations to preferences · 6.2 the reward model · 6.3 policy optimisation: PPO & the KL leash · 6.4 the limits of RLHF · 6.5 lab (Colab).

What we'll cover

Human comparisons can provide useful feedback when demonstrations are difficult to write. We introduce the RLHF pipeline and ask when human approval tracks the behaviour we want.

Mandatory readings

• Christiano et al. (2017), "Deep Reinforcement Learning from Human Preferences" (NeurIPS 2017; arXiv:1706.03741): read §2 (Preliminaries and Method), which sets up the problem and gives the comparison-based method in full. Skim the experiments for the Atari and simulated-robotics results. ≈1,700 words.

• OpenAI (2017), "Learning from human preferences" (openai.com): the friendlier way in, with the backflip demo showing a human choosing between two clips of the same agent. Read it alongside §2, not instead of it: the post gives you the idea in a minute, the section gives you the method. ≈850 words.

Total mandatory load: ≈2,550 words.

Optional readings

• Perrigo (2023), "Exclusive: OpenAI Used Kenyan Workers on Less Than $2 Per Hour" (TIME): the source behind "The people in the loop" above. Read it if you want the detail the box compresses.

• Stiennon et al. (2020), "Learning to summarize from human feedback" (arXiv:2009.01325): the bridge from control tasks to language, with unusually careful data-collection detail.

• Bai et al. (2022), "Training a Helpful and Harmless Assistant with RLHF" (arXiv:2204.05862): RLHF as actually deployed, at length.

Judging is easier than producing

Recall the asymmetry from Session 5.2: for most of what we want from an assistant, judging is far easier than producing.

You may be able to compare two answers without being able to write an ideal answer yourself. Ordinary SFT directly rewards a supplied target; a preference objective can use the comparison itself. Choosing preferred samples as SFT targets is also possible. The distinction concerns how feedback enters the objective, not a theorem that supervised models cannot outperform their demonstrators.

A short primer on reinforcement learning

If you have not met reinforcement learning before, this is what the rest of Session 6 assumes. Nothing in this box is about language models. The translation to them comes at the end.

In supervised learning every training example carries the right answer and the model is asked to reproduce it. Reinforcement learning has no right answers. An agent observes the state of its environment, chooses an action, and the environment responds with a new state and a number, the reward. Nothing says what the agent should have done. It learns by trying things and doing more of whatever scores well. Session 3.2 set this up formally as a Markov decision process.

Take a program learning to play a board game. The state is the position, the actions are the legal moves, and the reward is $+1$ for a win and $-1$ for a loss, delivered once, at the end. Two difficulties appear immediately. The reward is sparse and delayed: fifty moves produce a single number, long after most of them were played. And that number has to be shared out among those fifty moves, which is the problem of credit assignment. The game was lost; nothing tells you which move lost it.

The vocabulary is small.

Policy: the rule that picks an action in a given state, usually as a probability spread over the available actions. This is the thing being learned.
Episode: one complete run from start to finish. One game.
Return: the total reward collected over an episode.
Value, and the advantage built from it. Value is how well the agent does from a given position on average, under its current policy. Value matters because "was that good?" is the wrong question: winning from a position you were already expected to win from teaches nothing. The useful question is whether the outcome beat expectation, and the gap between what happened and what was expected is the advantage. A second model, the critic, is trained to supply the expectation.

The update follows from that: make actions that turned out better than expected more likely, and actions that turned out worse less likely. Because "better than expected" is itself estimated from a handful of episodes, a large step on that estimate can wreck a policy that was working. Keeping each step small is the problem PPO solves in 6.3.

Now the translation. This session is that same loop with the board game swapped out.

In reinforcement learning	In this session
agent	the language model being trained
state	the prompt, plus whatever has been generated so far
action	emitting the next token
episode	one prompt and one complete response
reward	a single number for the finished response, from the reward model of 6.2

Both difficulties transfer intact. A response of two hundred tokens earns one score, arriving only at the end, with nothing to say which token earned it. And the reason to take on that trouble, when supervised fine-tuning is so much simpler, is the asymmetry above: reinforcement learning needs only to score an answer, where supervised learning needs someone to have produced one.

The idea: optimise for what humans prefer

The idea

Instead of "imitate this answer", RLHF says: collect human judgements of which of two model outputs is better, distil those judgements into a learned reward model, and then use reinforcement learning to push the model toward outputs the reward model scores highly. Comparisons are cheaper and more reliable than demonstrations; the reward model turns a pile of "A > B" judgements into a signal you can optimise against at scale; and the result can exceed the quality of any single human demonstration, because it is chasing "better" rather than "copy this".

Preference training can improve responses beyond the supplied demonstrations when the feedback recognises those improvements and optimisation can find them. Supervised learning can also generalise beyond individual demonstrations. Both approaches can fail when their training signal rewards plausible-looking errors.

Comparisons versus demonstrations

For a paper-summary task, a demonstration asks an annotator to write a summary; a comparison asks them to choose between supplied summaries. Comparisons may be easier to collect, but their cost and reliability depend on the task and raters. Evaluate agreement and factual accuracy rather than assuming that preference labels are less noisy.

The preference-collection loop

The collection loop is where human values quietly enter the system. For a prompt $x$, sample two or more responses from the current model; show them to a human annotator with a set of instructions ("prefer helpful, honest, harmless answers; penalise confident falsehoods…"); record which they pick. Repeat across tens of thousands of prompts to build a dataset of triples $(x, y_w, y_l)$: a prompt, a preferred ("winning") response, and a dispreferred ("losing") one. Two choices here do a great deal of work, and both return in 6.2 and Session 8: who the annotators are (their language, context and training), and what the instructions told them to value. A reward model can only learn the preferences these choices wrote into the data.

The people in the loop

The paragraph above asks who the annotators are and answers in terms of what that does to the data. There is a second answer, about what the work does to them.

Much of this labelling is outsourced. In January 2023 TIME reported that OpenAI had contracted Sama, a San Francisco firm, to help build a filter for detecting toxic content, and that the work was done by employees in Kenya taking home between $1.32 and $2 an hour while OpenAI paid Sama $12.50 an hour for it (Perrigo, 2023). They read and categorised tens of thousands of passages describing violence, hate speech and sexual abuse. All four workers interviewed described being mentally scarred by it; counselling was available and they reported it was not usable under the productivity targets. Sama ended the contract eight months early.

The labelling existed to build the filter. This is not a general cost of building AI systems that happened to land on someone: the dataset existed to make the model safer, so the harm was incurred in the production of safety itself.

Hold both halves of that when 7.1 costs the same labour out, at roughly 830 hours per 100,000 comparisons, and treats it as a reason to move from human feedback to AI feedback. Fewer people having to read this material is a real argument for RLAIF. It is also an argument for removing the humans whose judgement the method exists to capture. Session 8.3 gives a framework for holding both at once, and 20.2 puts it in the wider critique of what "safety" is taken to cover.

A short history

From game-playing to chatbots

Christiano et al. (2017), "Deep RL from Human Preferences". The foundational paper: learn a reward function from human preferences between short trajectory segments, then optimise it with RL. It taught simulated robots and Atari agents complex behaviours from feedback on under 1% of their interactions; sometimes these were behaviours no one could easily hand-specify a reward for.
Stiennon et al. (2020), "Learning to summarize from human feedback". The first big language application: a reward model trained on human comparisons of summaries, then RL. The summaries were preferred by humans over the reference (human-written) summaries and over much larger supervised models. Optimising the learned reward beat optimising ROUGE, a hand-coded proxy.
Ouyang et al. (2022), InstructGPT. RLHF applied to general instruction-following, the recipe behind ChatGPT-style assistants. The central result: outputs from the 1.3B-parameter InstructGPT were preferred to those of the 175B-parameter GPT-3, a 100× smaller model made more useful by alignment rather than scale.
Bai et al. (2022), Anthropic's "Helpful and Harmless" assistant. RLHF for helpfulness and harmlessness together, with iterated (weekly) online data collection; found the "alignment tax" on raw capability to be small at scale.

RLHF as deployed alignment

RLHF is the technique that turned capable-but-unusable base models into the assistants now deployed to hundreds of millions of people. Its successes (InstructGPT) and its failure modes (6.4) therefore carry real stakes: they belong to the alignment method in production. Understanding what it optimises is understanding what today's "aligned" models are aligned to.

A common misconception: RLHF makes the model "smarter"

Preference-based RLHF changes which responses a model produces. InstructGPT’s 1.3B model was preferred to the separate 175B GPT-3 model in the reported comparison; the smaller model was fine-tuned from a base model of its own size. This demonstrates the value of post-training for that evaluation, not the absence of any capability change. Distinguishing improved elicitation from new task-solving ability requires broader, controlled tests.

The three-step pipeline

The three steps

Supervised fine-tuning: start from an SFT model (Session 5.2); this is the policy we will refine and the anchor we will stay close to.
Reward modelling (6.2): collect human comparisons of model outputs and fit a reward model $r_\phi$ that scores a response, via the Bradley–Terry model.
RL fine-tuning (6.3): optimise the policy to maximise $r_\phi$, penalised for drifting too far from the SFT model (the KL "leash"), typically with PPO, or skip the explicit RL with DPO.

Each step is an opportunity and a hazard. Step 2 compresses the entirety of "what humans want" into a single learned number: a proxy. Step 3 then optimises hard against that proxy, which (Session 3.1) is the recipe for specification gaming unless something holds it back. The KL leash is that something; whether it is enough is the question of 6.4.

Questions to bring to class

Give a task where you could confidently say "answer A is better than B" but could not have written the ideal answer yourself. Why is that gap the motivation for RLHF?
RLHF can in principle exceed human demonstration quality. State the precondition that makes this possible, and the precise way that same precondition can backfire.
InstructGPT (1.3B) beat GPT-3 (175B) on human preference. What does that tell you about where "usefulness" actually comes from, and what it does not tell you about safety?
The pipeline replaces "imitate a human" with "optimise a model of human approval". List the assumptions hidden in that substitution.

Step 2 is where human judgements become a number. Sub-session 6.2 builds the reward model (the Bradley–Terry preference model and its loss) and names the proxy we are about to optimise.