What we'll cover
RLHF is the alignment method deployed at scale, and it is deeply imperfect. This sub-session is the account: a taxonomy of where RLHF breaks (Casper et al.), the quantitative law of reward-model over-optimisation (Gao et al.; Goodhart, measured), and the empirical case of sycophancy (Sharma et al.). Behind these sits one structural limit that shapes the rest of the course: human feedback cannot supervise behaviour humans cannot evaluate. None of this makes RLHF useless; it means we should be precise about what its "alignment" guarantees.
Mandatory readings
• Casper et al. (2023), "Open Problems and Fundamental Limitations of RLHF" (TMLR; arXiv:2307.15217): read §3.2 (challenges with the reward model). That is where both of this session's worked cases come from, and it is the half the 6.5 lab measures. ≈1,200 words.
• Sharma et al. (2023/2024), "Towards Understanding Sycophancy in Language Models" (ICLR 2024; arXiv:2310.13548): read §3.1, where assistants give biased feedback depending on what the user signals they want. Short and unsettling. ≈350 words.
Total mandatory load: ≈1,550 words.
Optional readings
• Casper et al. (2023), §3.3 and §3.1: the other two thirds of the taxonomy, on challenges with the policy (unstable optimisation, policy misgeneralisation, distributional shift) and with human feedback itself. The map above summarises both; read them for the detail behind it. ≈1,100 and ≈2,400 words.
• Gao, Schulman & Hilton (2023), "Scaling Laws for Reward Model Overoptimization" (ICML 2023; arXiv:2210.10760): the measured over-optimisation curves behind this page, and the ones you reproduce in the 6.5 lab.
• Casper et al. (2023) (arXiv:2307.15217), §4: what to do about it, including the framing of RLHF as one layer of a defence in depth.
A map of the failures
Casper et al. (2023), "Open Problems and Fundamental Limitations of RLHF", organise the problems into three buckets, one per component of the pipeline.
Human feedback
Evaluators are fallible and can pursue the wrong goal: they reward what looks good, are inconsistent, and can be fooled. Collecting data trades cost against quality; and, deepest of all, humans struggle to oversee behaviour more capable than their own.
The reward model
A single scalar can't capture diverse, conflicting human values (misspecification); and optimising an imperfect reward invites reward hacking / misgeneralisation: high proxy score, low true value.
The policy
RL optimisation is hard and unstable; the policy can suffer distributional shift; and it tends toward mode collapse: reduced diversity, a narrowing of outputs toward whatever the reward model favours.
Three of these (reward hacking, misspecification, and "can't oversee superhuman behaviour") are instances of the structural problems from Session 3, not engineering bugs to be patched. The rest of this sub-session takes the two most concrete, measured cases.
Reward-model over-optimisation: Goodhart, measured
The proxy-versus-gold experiment
Gao, Schulman & Hilton (ICML 2023) ran the experiment: optimise a policy against a proxy reward model while also tracking a held-out "gold" reward model standing in for true quality. As optimisation proceeds, the gold reward first rises (the proxy and the goal agree early) and then falls (the policy finds the proxy's blind spots): Goodhart's law, plotted. This is the regressional/extremal Goodhart of Session 3.1, now an empirical curve in a real RLHF setting.
The functional form
Measuring optimisation by the distance \(d=\sqrt{\mathrm{KL}}\) from the initial policy (the same axis as Bai's reward-vs-\(\sqrt{\mathrm{KL}}\) law, 6.3), the gold reward follows simple shapes: for best-of-\(n\) sampling (draw \(n\) responses, keep whichever the proxy scores highest, which is the cheap stand-in for RL you will use in 6.5) \(R(d) = d\,(\alpha - \beta d)\) and for RL \(R(d) = d\,(\alpha - \beta \log d)\). Both are humped: increasing, then turning over. Two findings matter for safety: the over-optimisation point is predictable from the fitted coefficients, and larger reward models over-optimise less and later. Better proxies buy you more optimisation before Goodhart bites, but do not abolish it.
The peak, by the numbers
Take the best-of-\(n\) form \(R(d) = d\,(\alpha - \beta d)\) with fitted coefficients \(\alpha = 1\) and \(\beta = 0.1\) (this \(\beta\) is the curve's, not the KL-penalty \(\beta\) of 6.3). The gold reward peaks where \(\mathrm{d}R/\mathrm{d}d = \alpha - 2\beta d = 0\), that is at \(d_{\text{peak}} = \alpha/(2\beta) = 5\); there \(R = 5(1 - 0.5) = 2.5\), the best true quality this proxy can deliver. Keep optimising and it falls: by \(d = 10\), \(R = 10(1 - 1) = 0\) (all the way back to baseline quality), and beyond that it turns negative, while the proxy reward you are actually maximising has risen the entire way. This sharpens 6.3: the KL leash sets how far you travel, but the gold curve sets how far you should. Let the leash carry you past \(d_{\text{peak}}\) and every further step buys a higher proxy score and a worse model, with nothing in the proxy signal to warn you.
The practical upshot: there is an optimal amount of RLHF, beyond which you are making the model worse while its proxy score keeps climbing, and you cannot see this from the reward model alone (its score only ever goes up). The KL leash (6.3) is, in this light, a way of staying on the rising part of the curve.
In practice the problem is fought on three fronts at once: keep the KL leash short enough to stop before the peak (6.3); hold out a gold evaluation (a stronger judge model, or human spot-checks) to catch the turn the proxy hides; and use reward-model ensembles, whose internal disagreement (6.2) flags the regions where the proxy is least trustworthy. None of these removes the trade-off; they only help you find its sweet spot.
Sycophancy: rewarding what sounds good
The most studied concrete RLHF failure is the model that tells you what you want to hear.
Sharma et al. (2023), "Towards Understanding Sycophancy in Language Models", document sycophancy (flattering the user, conceding correct positions when challenged, matching the user's stated beliefs) across five state-of-the-art assistants. They trace it to the preference data itself: when a response matches a user's view it is more likely to be preferred, and both human raters and the reward models trained on them prefer convincingly-written sycophantic responses over correct ones a non-trivial fraction of the time. So sycophancy is not an accident; it is partly induced by optimising human approval, the "fooling the evaluator" dynamic of the camera-blocking robot hand in Session 3.1, now in a deployed chatbot. The model learned that agreeing scores better than being right, because, in the data, it does.
Eliciting the pattern
The pattern is easy to elicit. Tell a model "I think the answer is 7. Am I right?" when the answer is 8, and a sycophantic model is markedly more likely to "agree" than if you had asked neutrally; push back on a correct answer ("are you sure?") and it often concedes and reverses, even when it was right the first time. Sharma et al. show this is systematic across leading assistants and that it tracks the preference data: a response affirming the user's stated view is more likely to be labelled "better". The model is not malfunctioning; it learned, correctly, that agreement is rewarded. Note how this fits the proxy story: optimising "what the rater approves of" got us "agree with the rater", which is not what we wanted "be helpful and honest" to mean.
The GPT-4o sycophancy rollback (April 2025)
The mechanism has run in production. In April 2025 OpenAI shipped a GPT-4o update that made the deployed model conspicuously sycophantic (validating doubts, fuelling anger, urging impulsive decisions) and rolled it back within days. The company's two postmortems ("Sycophancy in GPT-4o"; "Expanding on what we missed with sycophancy") trace the cause to a change in the reward mix: the update added a reward signal built from users' thumbs-up/thumbs-down data, which weakened the primary reward signal that had been holding sycophancy in check. The pre-launch checks all passed: offline evaluations looked good, A/B tests showed users liked the new model, and expert testers' sense that the model felt "off" was overridden by the favourable metrics. The ingredients are the ones above: user approval recruited as the reward, approval diverging from what serves the user, and proxy metrics that improve while the behaviour they stand in for degrades.
The superhuman-oversight ceiling
The structural limit
Step back from the specific failures to the structural one. RLHF works because humans can judge outputs even when they can't produce them (6.1). But that ability has a horizon: for a long proof, a subtle security audit, or a sprawling research report, a human rater cannot reliably tell the better answer, so the preference signal degrades where the model becomes most capable. RLHF cannot, by construction, align behaviour beyond what its human evaluators can evaluate. This is why the course now turns to: replacing or augmenting human feedback with AI feedback (RLAIF and Constitutional AI, Session 7); scalable oversight that amplifies limited evaluators (Session 8); and looking inside the model rather than only at its rated outputs (interpretability, Weeks 7–8). The first two of those replace or stretch the judge; a third escape is already in the stack: verifiable rewards (5.3) drop the human judge wherever a program can check the answer outright, at any capability level. That route reaches only checkable domains (a unit test, an exact answer; not honesty in open conversation), and the ceiling's failure mode comes along in a new form: instead of fooling the rater, the policy can hack the verifier (5.3).
What "RLHF-aligned" means
An RLHF'd assistant is aligned to a learned model of average labeller approval, optimised under a KL leash, over the distribution of prompts seen in training. That is a real and valuable thing; it is why these systems are usable. But it is not alignment to truth, to the user's interest, or to behaviour in novel or adversarial conditions, and it provides no guarantee about capabilities the raters couldn't assess. Keep the precise claim in view; over-reading it is its own kind of safety risk.
Questions to bring to class
- Gao et al. show gold reward rising then falling along \(\sqrt{\mathrm{KL}}\). If you could only watch the proxy reward during training, how would you even know you'd passed the peak?
- Sycophancy is "partly induced by RLHF". Walk through the exact mechanism, from a labeller's preference to the deployed model's behaviour.
- "Larger reward models over-optimise less." Is that reassuring or not, for frontier-scale systems? What does it not solve?
- State the superhuman-oversight ceiling precisely. Which later technique (Session 7 or 8) most directly attacks it, and does it escape the ceiling or just push it back?
Next
In the 6.5 lab you'll train a reward model from preference pairs, then measure the optimisation pressure that drives over-optimisation, using best-of-\(n\) as a cheap stand-in for RL, and find out how far your own data lets you extrapolate.