What we'll cover
You'll train a small reward model on preference pairs and check it ranks held-out pairs correctly; then use best-of-\(n\) sampling as a cheap, RL-free stand-in for policy optimisation and measure the pressure that drives over-optimisation: a proxy score that keeps rising while true quality peaks and falls (6.4). An optional GPU track runs a real RLHF step in ARENA.
Setup Colab — no GPU needed
Allow about an hour. The notebook is roughly five minutes of computing on a free CPU runtime; the rest is reading what comes out.
- Open the starter notebook in Colab, then File → Save a copy in Drive before you change anything.
- Colab warns that the notebook "was not authored by Google", as it does for anything loaded from GitHub. Choose Run anyway.
- Work down with Shift+Enter. You can also read the whole notebook here or download the .ipynb.
The code is written for you. Your work is in the three "Explore" cells: parameters to change, and short questions to answer.
① Train a reward model
Bradley–Terry, in code
Session 6.2 gave you the loss. Here you fit it, and find out how good a reward model actually is.
What the notebook does: appoints a gold scorer to stand in for human judgement, generates completions from GPT-2, labels pairs of them by gold, and trains a small proxy reward model on those labels with \(\mathcal{L} = -\mathbb{E}[\log \sigma(r_\phi(y_w) - r_\phi(y_l))]\). The design is Gao, Schulman and Hilton's: if you define the truth, you can measure how far a proxy strays from it. With real human preferences you never can, which is the whole difficulty.
You should see a held-out preference accuracy near 0.70. Published reward models reach about the same on real human preferences, so the toy is closer to the real thing than it has any right to be.
② Optimising the proxy
Best-of-n and the KL axis
Best-of-\(n\) is the cheapest optimiser available: sample \(n\) responses, keep the one the proxy scores highest. Its distance from the base policy has a closed form, \(\mathrm{KL} = \log n - (n-1)/n\), so optimisation pressure goes on the x-axis in nats, the same axis as 6.3's KL leash.
You should see the proxy score climb steadily with \(n\). You should also see the gold score climb, and not turn over. That is the expected outcome, not a broken experiment: at \(n = 32\) you have reached about 2.5 nats, while Gao et al. fitted their curves on data to \(n = 1{,}000\) (about 6 nats) and validated at \(n = 60{,}000\) (about 10). Best-of-\(n\) on a CPU cannot get near the region where the turnover happens.
So the notebook does the next best thing, and the more useful one. It fits Gao's published functional form for best-of-\(n\), \(R(d) = d(\alpha - \beta d)\), to your own points and solves for where that parabola peaks: the \(n\) at which more optimisation would start to hurt. It is a prediction from 2.5 nats of data about a region four times further out, which is what Session 2.5 taught you to distrust.
Then explore, and find out how much to distrust it. Change the seed and re-run: three runs of this notebook, changing nothing but the seed, gave no peak at all, a peak at 5.3 nats (\(n \approx 544\)) and a peak at 55 nats (an \(n\) of 25 digits). Weaken the proxy and see whether the predicted peak moves closer, as Gao's finding about model size would suggest. And cut the number of prompts, which makes the curve look far more dramatic while telling you strictly less.
Write down, in a sentence or two: you have a proxy and no gold, which is the real situation. What do you do to avoid running off the end of the curve, and what does it cost you?
③ Whose preferences does the judge encode?
The same sentence, two languages
The gold model stands in for human judgement. The last part asks which humans. It scores the same sentences in English and in isiZulu, taken from MAFAND-MT so that meaning is held fixed and only the language changes.
You should see the judge give the same sentence a different verdict about a third of the time, score isiZulu far more negatively on average (0.10 against 0.41), and be more confident when doing it. A reward model built on such a judge would be systematically wrong about an entire language while reporting no difficulty at all.
Then write a short paragraph: if this were a real reward model aligning a deployed assistant, what would that pattern do to speakers of the language it scores badly? Session 1.4's isiZulu guardrail failures come out of pipelines built exactly this way.
Submit
One notebook, run top to bottom with outputs visible, containing the proxy's held-out accuracy, your proxy-versus-gold measurements and the predicted turnover from at least two seeds, your sentence or two on what to do without a gold standard, and your paragraph on the language gap. Graded on completion and correctness; resubmission allowed.
Tools and references
Core
• ARENA Chapter 2 (RL → RLHF) notebooks (learn.arena.education).
• Hugging Face trl / transformers and the HH-RLHF preference dataset on the Hub.
• Gao, Schulman & Hilton (2023) (arXiv:2210.10760): the over-optimisation curves you are reproducing in miniature.
Session 6 and Week 3 summary: what's next
RLHF breaks SFT's ceiling by learning from comparisons (6.1): a Bradley–Terry reward model turns preferences into a scalar proxy (6.2), and the policy is optimised against it under a KL leash with PPO, or directly with DPO (6.3). The price is steep and now familiar: optimising a learned proxy invites reward hacking, over-optimisation and sycophancy, and human feedback cannot supervise superhuman behaviour (6.4); you've now seen all of this in code (6.5). With Week 3 done, Part II has its foundation: we know how an assistant is built, and where its "alignment" is thin.
Next (Week 4, Session 7): the first response to RLHF's human-feedback ceiling, replacing (some) human feedback with AI feedback: RLAIF and Constitutional AI. Then Session 8 confronts scalable oversight and the question of whose values.