From demonstrations to preferences

A short primer on reinforcement learning

If you have not met reinforcement learning before, this is what the rest of Session 6 assumes. Nothing in this box is about language models. The translation to them comes at the end.

In supervised learning every training example carries the right answer and the model is asked to reproduce it. Reinforcement learning has no right answers. An agent observes the state of its environment, chooses an action, and the environment responds with a new state and a number, the reward. Nothing says what the agent should have done. It learns by trying things and doing more of whatever scores well. Session 3.2 set this up formally as a Markov decision process.

Take a program learning to play a board game. The state is the position, the actions are the legal moves, and the reward is \(+1\) for a win and \(-1\) for a loss, delivered once, at the end. Two difficulties appear immediately. The reward is sparse and delayed: fifty moves produce a single number, long after most of them were played. And that number has to be shared out among those fifty moves, which is the problem of credit assignment. The game was lost; nothing tells you which move lost it.

The vocabulary is small.

• Policy: the rule that picks an action in a given state, usually as a probability spread over the available actions. This is the thing being learned.
• Episode: one complete run from start to finish. One game.
• Return: the total reward collected over an episode.
• Value, and the advantage built from it. Value is how well the agent does from a given position on average, under its current policy. Value matters because "was that good?" is the wrong question: winning from a position you were already expected to win from teaches nothing. The useful question is whether the outcome beat expectation, and the gap between what happened and what was expected is the advantage. A second model, the critic, is trained to supply the expectation.

The update follows from that: make actions that turned out better than expected more likely, and actions that turned out worse less likely. Because "better than expected" is itself estimated from a handful of episodes, a large step on that estimate can wreck a policy that was working. Keeping each step small is the problem PPO solves in 6.3.

Now the translation. This session is that same loop with the board game swapped out.

Both difficulties transfer intact. A response of two hundred tokens earns one score, arriving only at the end, with nothing to say which token earned it. And the reason to take on that trouble, when supervised fine-tuning is so much simpler, is the asymmetry above: reinforcement learning needs only to score an answer, where supervised learning needs someone to have produced one.

The preference-collection loop

The collection loop is where human values quietly enter the system. For a prompt \(x\), sample two or more responses from the current model; show them to a human annotator with a set of instructions ("prefer helpful, honest, harmless answers; penalise confident falsehoods…"); record which they pick. Repeat across tens of thousands of prompts to build a dataset of triples \((x, y_w, y_l)\): a prompt, a preferred ("winning") response, and a dispreferred ("losing") one. Two choices here do a great deal of work, and both return in 6.2 and Session 8: who the annotators are (their language, context and training), and what the instructions told them to value. A reward model can only learn the preferences these choices wrote into the data.

From game-playing to chatbots

• Christiano et al. (2017), "Deep RL from Human Preferences". The foundational paper: learn a reward function from human preferences between short trajectory segments, then optimise it with RL. It taught simulated robots and Atari agents complex behaviours from feedback on under 1% of their interactions; sometimes these were behaviours no one could easily hand-specify a reward for.
• Stiennon et al. (2020), "Learning to summarize from human feedback". The first big language application: a reward model trained on human comparisons of summaries, then RL. The summaries were preferred by humans over the reference (human-written) summaries and over much larger supervised models. Optimising the learned reward beat optimising ROUGE, a hand-coded proxy.
• Ouyang et al. (2022), InstructGPT. RLHF applied to general instruction-following, the recipe behind ChatGPT-style assistants. The central result: outputs from the 1.3B-parameter InstructGPT were preferred to those of the 175B-parameter GPT-3, a 100× smaller model made more useful by alignment rather than scale.
• Bai et al. (2022), Anthropic's "Helpful and Harmless" assistant. RLHF for helpfulness and harmlessness together, with iterated (weekly) online data collection; found the "alignment tax" on raw capability to be small at scale.

The three steps

1. Supervised fine-tuning: start from an SFT model (Session 5.2); this is the policy we will refine and the anchor we will stay close to.
2. Reward modelling (6.2): collect human comparisons of model outputs and fit a reward model \(r_\phi\) that scores a response, via the Bradley–Terry model.
3. RL fine-tuning (6.3): optimise the policy to maximise \(r_\phi\), penalised for drifting too far from the SFT model (the KL "leash"), typically with PPO, or skip the explicit RL with DPO.