What we'll cover
You'll train a small reward model on preference pairs and check it ranks held-out pairs correctly; then use best-of-\(n\) sampling as a cheap, RL-free stand-in for policy optimisation and measure the pressure that drives over-optimisation: a proxy score that keeps rising while true quality peaks and falls (6.4). An optional GPU track runs a real RLHF step in ARENA.
Setup Colab — no GPU needed
Allow about an hour. Allow around ten minutes for computing on a standard Colab CPU runtime; downloads and runtime load can change this. The evaluation-generation cell alone took about six minutes in the 11 September 2026 check. The rest of the hour is for reading the outputs and exploring.
- Open the starter notebook in Colab, then File → Save a copy in Drive before you change anything.
- Colab warns that the notebook "was not authored by Google", as it does for anything loaded from GitHub. Choose Run anyway.
- Work down with Shift+Enter. You can also read the whole notebook here or download the .ipynb.
The code is written for you. Your work is in the four "Explore" cells: parameters to change, and short questions to answer.
① Train a reward model
Bradley–Terry, in code
Session 6.2 gave you the loss. Here you fit it, and find out how good a reward model actually is.
What the notebook does: appoints a gold scorer to stand in for human judgement, generates completions from GPT-2, labels pairs of them by gold, and trains a small proxy reward model on those labels with \(\mathcal{L} = -\mathbb{E}[\log \sigma(r_\phi(y_w) - r_\phi(y_l))]\). The design is Gao, Schulman and Hilton's: if you define the truth, you can measure how far a proxy strays from it. With real human preferences you never can, which is the whole difficulty.
Measure held-out preference accuracy against the toy scorer’s labels. This is a different dataset and judgement task from published human-preference evaluations; a similar accuracy would not make the models equivalent.
② Optimising the proxy
Best-of-n and the KL axis
Best-of-\(n\) selects the highest-scoring response among \(n\) samples. The expression \(\log n-(n-1)/n\) gives a KL reference for idealised sampling with continuous reward ranks. This notebook resamples a finite candidate pool, so use that axis as an approximation and inspect ties and pool size.
Plot what happens to both scores as \(n\) increases. With this small candidate pool, gold may keep rising. A downturn is not required for a successful submission. The absence of one here does not show that larger searches are safe, or that a downturn must occur at some particular larger \(n\).
The notebook fits Gao’s best-of-\(n\) functional form, \(R(d)=d(\alpha-\beta d)\), to your points and computes a possible peak. Treat that peak as an extrapolation conditional on the fitted shape. Repeat the fit with different seeds and report whether the predicted peak is stable or supported by the observed range.
Then explore, and find out how much to distrust it. Change the seed and re-run: three runs of this notebook, changing nothing but the seed, gave no peak at all, a peak at 5.3 nats (\(n \approx 544\)) and a peak at 55 nats (an \(n\) of 25 digits). Weaken the proxy and see whether the predicted peak moves closer, as Gao's finding about model size would suggest. And cut the number of prompts, which makes the curve look far more dramatic while telling you strictly less.
Write down, in a sentence or two: you have a proxy and no gold, which is the real situation. What do you do to avoid running off the end of the curve, and what does it cost you?
③ Whose preferences does the judge encode?
The same sentence, two languages
The gold model is an English movie-review sentiment classifier. Compare its scores on parallel English and isiZulu news sentences from MAFAND-MT. Both inputs are truncated to 64 tokens, which can retain different amounts of content. The dataset supplies translations, not ground-truth sentiment labels.
Report prediction agreement, mean scores and confidence separately. Disagreement alone does not identify which prediction is wrong. Use labelled examples, checked translations and complete short sentences to investigate language comprehension, domain shift and truncation.
Write a short paragraph separating your observations from hypotheses about harm in deployment. Propose a controlled evaluation that could establish whether the judge makes more sentiment errors in one language.
Submit
One notebook, run top to bottom with outputs visible, containing the proxy's held-out accuracy, your proxy-versus-gold measurements and the predicted turnover from at least two seeds, your sentence or two on what to do without a gold standard, and your paragraph on the language gap. Graded on completion and correctness; resubmission allowed.
Tools and references
Core
• ARENA Chapter 2 (RL → RLHF) notebooks (learn.arena.education).
• Hugging Face trl / transformers and the HH-RLHF preference dataset on the Hub.
• Gao, Schulman & Hilton (2023) (arXiv:2210.10760): the gold-versus-proxy design and the curve whose extrapolation the notebook tests.
Session 6 and Week 3 summary: what's next
RLHF uses preference comparisons to learn a scalar reward, then optimises against it; DPO uses comparisons directly. These methods can improve behaviour while retaining or amplifying errors in the feedback. The lab illustrates proxy optimisation and the limits of an extrapolated curve. Its multilingual extension asks what additional evidence would be needed to validate the scorer outside its training domain.
Next (Week 4, Session 7): the first response to RLHF's human-feedback limitations, replacing (some) human feedback with AI feedback: RLAIF and Constitutional AI. Then Session 8 confronts scalable oversight and the question of whose values.