Session 6.2 — The reward model

What we'll cover

The reward model is the hinge of RLHF: it converts a pile of "response A is better than B" judgements into a function \(r_\phi(x,y)\) that scores any response, which the policy can then be optimised against. This sub-session derives that function from the classic Bradley–Terry model of paired comparisons, writes down its training loss, and names what it is: a learned, scalar proxy for human approval, with all the Goodhart exposure of Session 3.1 built in.

Mandatory readings

• Ouyang et al. (2022), InstructGPT (arXiv:2203.02155): read §3.5 (Models). You read §3.1 to §3.4 in Session 5.2; this is the part that trains the reward model, and the loss on this page is theirs. ≈1,000 words.

Total mandatory load: ≈1,000 words.

Optional readings

• Bradley, R. A. & Terry, M. E. (1952), "Rank Analysis of Incomplete Block Designs: I" (Biometrika 39(3/4), 324–345): the original paired-comparison model, seventy years before anyone applied it to a chatbot.

• Stiennon et al. (2020) (arXiv:2009.01325): reward modelling for summarisation, with the data-collection detail most papers omit.

From comparisons to a score

We collect data of one shape: a prompt \(x\), two candidate responses, and a human's judgement of which is better.

Write the preferred ("winning") response \(y_w\) and the dispreferred ("losing") one \(y_l\). We want a scalar reward function \(r_\phi(x,y)\), a neural network with parameters \(\phi\), such that better responses get higher scores. The question is how to turn discrete "A beats B" judgements into a continuous score. The standard answer borrows a seventy-year-old model from statistics.

The Bradley–Terry model

Bradley & Terry (1952) modelled paired comparisons by giving each item a latent "worth" and making the probability that one beats another a function of the difference in worth. Identifying "worth" with our reward, the probability a human prefers \(y_w\) to \(y_l\) is the logistic of the reward gap:

\[ P(y_w \succ y_l \mid x) = \sigma\!\big( r_\phi(x, y_w) - r_\phi(x, y_l) \big), \qquad \sigma(z) = \frac{1}{1 + e^{-z}}. \]

So if the two responses score equally the human is a coin-flip to prefer either; the larger the reward gap, the more confidently the better-scoring one should be preferred. Only differences in reward matter: the model has no absolute zero, which is fine, because we only ever optimise relative preferences.

The logistic, in numbers

The Bradley–Terry probability depends only on the reward gap \(\Delta = r_\phi(x,y_w) - r_\phi(x,y_l)\): a gap of \(0\) gives \(\sigma(0)=0.5\) (a coin-flip; the model is indifferent), a gap of \(1\) gives \(\sigma(1)\approx 0.73\), and a gap of \(2\) gives \(\sigma(2)\approx 0.88\). So the reward scale is calibrated in log-odds of preference: training nudges the gap until the model's predicted preference probabilities match how often humans actually chose each response. A reward difference is therefore a statement about how reliably one response beats another.

A common misconception: the reward number is a quality score

It is not. A reward of 2.3 means nothing on its own; only differences within the same prompt are trained, so raw rewards are not comparable across prompts, and a high score is not a claim that an answer is correct or true, only that the average labeller would more likely prefer it to its rival. Two answers can both be wrong while one scores far higher. Reading the reward as "how good this is" rather than "how preferred this is" is the first step toward trusting it too much.

Training the reward model

Fit \(\phi\) by maximum likelihood: make the model assign high probability to the comparisons humans actually made.

Taking the negative log-likelihood of the Bradley–Terry model over a dataset \(\mathcal{D}\) of comparisons gives the reward-model loss:

\[ \mathcal{L}(\phi) = -\,\mathbb{E}_{(x,\,y_w,\,y_l)\sim\mathcal{D}}\Big[ \log \sigma\!\big( r_\phi(x, y_w) - r_\phi(x, y_l) \big) \Big]. \]

Minimising it pushes the reward of preferred responses above that of dispreferred ones, by a margin that grows with how consistently humans favoured them. In practice \(r_\phi\) is usually the SFT model itself with its final unembedding replaced by a single scalar "reward head", so the reward model starts from the same language understanding as the policy it will judge.

A worked training step, by hand

Take one comparison the human labelled \(y_w \succ y_l\), and suppose the current model scores \(r_\phi(x,y_w)=1.2\) and \(r_\phi(x,y_l)=0.8\). The gap is \(\Delta = 0.4\), so the model already predicts the human's choice with probability \(\sigma(0.4)\approx 0.60\), and this example contributes loss \(-\log 0.60 \approx 0.51\). The gradient of that loss raises \(r_\phi(x,y_w)\) and lowers \(r_\phi(x,y_l)\), widening the gap so that next time \(\sigma\) sits closer to 1. Had the model scored them the wrong way round (\(\Delta=-0.4\), so \(\sigma\approx 0.40\)), the loss would be larger (\(\approx 0.91\)) and the push correspondingly harder, until the preferred response outscores its rival. That is reward-model training: nudge the scores so each preferred response sits a comfortable margin above the other, with bigger nudges where the model is currently more wrong.

What the reward model measures

You now have a function that, given a prompt and a response, returns a number meant to track "how much a human would approve". That is enormously useful: it can score millions of responses automatically, far beyond the comparisons you collected. But notice what it is not. It is not a measure of truth, or of the user's real interest, or of long-run benefit. It is a learned model of the average labeller's snap judgement, compressed to one scalar. That is the object the policy is about to be optimised against, as hard as the KL leash allows.

Evaluating a reward model

The natural metric is preference accuracy: on held-out comparisons, how often does the reward model score the human-preferred response higher? A well-trained model lands comfortably above the 50% chance line but rarely near 100%, and it should not, because humans themselves disagree on hard pairs perhaps a quarter to a third of the time, so there is an irreducible noise floor it cannot beat without overfitting to one annotator's taste. Two consequences set up the rest of the session. The model is trustworthy only near the responses it was trained on; push the policy into unusual territory (6.3) and its judgements decay where you can least check them. And because a single network is a noisy estimate, labs sometimes train an ensemble and treat disagreement among its members as a cheap "here be dragons" signal. Both points are the groundwork for the over-optimisation of 6.4.

The reward model as proxy

Goodhart exposure by construction

Re-read Session 3.1 with \(r_\phi\) in the role of the proxy. We cannot write down "be helpful and honest", so we learn a stand-in from human comparisons, and then optimise it. Every gap between \(r_\phi\) and true human values is a place the optimiser can exploit: a response that scores high on the reward model without being good. The reward model is unbiased and useful in the region of responses it was trained on; push the policy into regions it never saw and the proxy and the goal come apart. This is what optimising a learned proxy means, not a flaw in any particular reward model.

Three built-in limitations

One scalar, plural values. Human preferences are diverse and multidimensional (helpful vs. harmless vs. honest can conflict). Collapsing them to a single number forces trade-offs the model makes silently, on whose behalf is unclear (Session 8's "whose values?").
It inherits the labellers. The reward model is only as representative as the comparison data, which is overwhelmingly English and drawn from particular annotator pools. Preferences in under-represented languages and contexts are barely in the signal (the Global-South thread again).
It rewards what looks good. Labellers judge from the response alone; they prefer confident, fluent, agreeable answers. So the reward model can score persuasiveness over correctness, the seed of sycophancy (6.4).

None of this is hypothetical. A reward model trained on labellers who liked thorough answers will reliably score a longer, more confidently-worded response above a terse correct one; optimise hard against it (6.3) and the policy discovers that, padding and hedging its way to a high reward without getting any more right. That is reward hacking in miniature, and it is why the next sub-session spends as much care on restraining the optimiser as on running it.

Questions to bring to class

Why does the Bradley–Terry model depend only on the difference \(r_\phi(x,y_w) - r_\phi(x,y_l)\)? What does that imply about the absolute scale of the reward?
The reward model is typically the SFT model with a scalar head. Why start there rather than from a fresh network?
Map the reward model onto Session 3.1's "proxy". Which of Goodhart's four variants do you expect to bite hardest when the policy is optimised against it, and why?
A single scalar must rank a helpful-but-blunt answer against a polite-but-evasive one. Who decided that trade-off, and where in the pipeline?

With a reward model in hand, sub-session 6.3 optimises the policy against it, and explains the KL "leash" that stops the optimiser from running off into the reward model's blind spots, plus PPO and the simpler DPO alternative.