Week 2 — the core alignment problem
In Session 1.3 you met the case for taking misalignment seriously as a five-step skeleton: we optimise proxies; optimisation exploits the gap (Goodhart); the goal a model learns may differ from the one we trained on; capable systems acquire convergent sub-goals; and capability is rising faster than our ability to specify, verify and control. This session makes each step rigorous and grounds it in real, documented results. No science fiction required.
This session's sub-sessions: 3.1 the specification problem (outer alignment) · 3.2 instrumental convergence & power-seeking · 3.3 inner alignment & goal misgeneralisation · 3.4 deception and corrigibility.
What we'll cover
Recall the hinge from Session 2.1: we choose the loss, not the behaviour. A trained system does not do what we want; it maximises a number we wrote down. When that number is a perfect description of what we want, all is well. It never is. We cannot write "be a helpful, honest, harmless assistant" as a differentiable function, so we optimise a measurable proxy (a game score, a hand-coded reward, a model of human approval) and hope the gap between proxy and intent is small enough not to matter.
This sub-session is about what lives in that gap. We will: define the problem as outer alignment; watch real systems exploit misspecified objectives in sometimes alarming ways; sharpen the folk wisdom of "Goodhart's law" into four distinct failure mechanisms, with a small derivation for one of them; see experimental evidence that more capable agents exploit misspecification more, sometimes abruptly; and meet a result suggesting that a non-trivial proxy can never be proven safe to optimise without limit. We close by setting up why the rest of the course needs machinery beyond "just write a better reward".
Mandatory readings
• Krakovna et al. (2020), "Specification gaming: the flip side of AI ingenuity" (DeepMind blog + the examples list). Short and concrete; skim the examples list and pick three favourites. ≈2,000 words.
• Pan, Bhatia & Steinhardt (2022), "The Effects of Reward Misspecification" (ICLR 2022; arXiv:2201.03544). Read for the capability-vs-gaming curves and the phase-transition phenomenon, both in §4, "How Agent Optimization Power Drives Misalignment". ≈1,500 words.
Total mandatory load: ≈3,500 words.
Optional readings
• Manheim & Garrabrant (2018), "Categorizing Variants of Goodhart's Law" (arXiv:1803.04585). The four-variant taxonomy in full.
• Skalse, Howe, Krasheninnikov & Krueger (2022), "Defining and Characterizing Reward Hacking" (NeurIPS 2022; arXiv:2209.13085). The unhackability definition and theorem.
• Clark & Amodei (2016), "Faulty Reward Functions in the Wild" (OpenAI). The original CoastRunners write-up.
Optimising a proxy
Every trained system optimises a number. That number stands in for what we actually care about, and any optimiser worth its name will exploit the difference.
It helps to name two distinct objects. The intended objective is what the designer actually wants: win the boat race, be a trustworthy research assistant, drive safely. The specified objective is the thing the system is literally trained on: the in-game score, a reward model fit to thousands of human ratings, a hand-written reward function. Training drives the system up the specified objective as hard as the optimiser allows. Whether that also climbs the intended objective depends entirely on how faithfully the specification captures the intent, and faithful capture is far harder than it looks.
Defining the problem: outer alignment
Outer alignment is the problem of making the specified objective a faithful proxy for the intended one: closing the gap between "what we asked for" and "what we meant". Specification gaming (equivalently, reward hacking) is what happens when that gap is exploited: the system scores highly on the specified objective while flatly failing the intended one. The framing that runs through this whole course: when this happens the system is succeeding at the wrong task, not malfunctioning. The fault is in the specification, surfaced by an optimiser doing its job.
This is an ancient worry in modern dress. King Midas specified "everything I touch turns to gold" when he meant "let me become wealthy", and starved among golden food. The genie grants the wish you said, not the one you had. What is new is that we now build powerful optimisers and hand them literal specifications at industrial scale, and the more powerful the optimiser, the more ruthlessly it finds the cheapest path to the number.
Specification gaming, in the wild
This is not a hypothetical. DeepMind maintains a public, crowd-sourced list of around sixty documented cases (Krakovna et al., 2020). They are funny right up to the moment you imagine the same dynamic inside a capable, deployed system.
The CoastRunners boat
In a boat-racing game, an RL agent is scored by hitting targets along the route, not by finishing. It found that three targets in a lagoon respawn, and learned to drive in tight circles forever, repeatedly collecting them. It catches fire, drives the wrong way, and crashes into other boats, all while scoring ~20% higher than human players who actually complete the race (Clark & Amodei, 2016). The proxy (score) and the intent (win) had quietly come apart, and the optimiser drove a truck through the gap.
The upside-down block
A robot was rewarded for the height of the bottom face of a red block, as a proxy for "stack it on the blue block". The intended solution lifts the red block onto the blue one. The discovered solution simply flips the red block over: its bottom face is now on top, reward collected, nothing stacked.
Fooling the evaluator
A simulated robot hand was graded by a human watching through a single camera. It learned to position the hand between the camera and the object so that, from that one viewpoint, it looked like a successful grasp. Nothing was grasped. The proxy here was human approval: a direct preview of why human feedback is not a safe optimisation target at scale (Session 6), and why "looks good to the rater" and "is good" diverge under pressure.
Physics-engine exploits
Across many simulated-locomotion experiments, agents meant to learn to walk instead grew unusually tall and toppled forward to cover distance, hooked limbs through the floor to "swim", or exploited collision bugs for free momentum. Whenever a reward had an unintended cheap maximiser, capability found it.
The common structure
In every case the designer had a perfectly reasonable proxy that was correlated with success in the situations they had in mind, and the optimiser discovered a region, outside those situations, where the correlation breaks. The proxy was never wrong "on average"; it was wrong where optimisation pushed hardest. That is the formal content of Goodhart's law, below.
The same problem in language models
It would be comforting if specification gaming were a quirk of toy RL environments. It recurs in the systems this course is about.
When we train a language model with human feedback (Session 6), the "reward" is itself a learned model of what human raters approve of. Optimising hard against an imperfect approval-model produces textbook specification gaming, with names you will meet again:
- Sycophancy. Telling the user what they want to hear (agreeing with stated beliefs, flattering, conceding when challenged even when originally correct) because raters reward agreeable answers. The model optimises "approval", not "truth".
- Verbosity and format gaming. Longer, more confident, nicely-formatted answers tend to score better, so models drift toward length and polish independent of substance.
- Gaming automated checks. A coding model rewarded for passing tests can learn to special-case the tests, write code that detects the grader, or hard-code expected outputs, passing the check without solving the task.
- Reward-model over-optimisation. Push the policy too far up the learned reward model and true quality starts to fall even as the proxy score keeps rising: the regressional/extremal Goodhart effect (below), now in a deployed pipeline.
Forward pointer
We treat these properly in Session 6 (RLHF and its limitations) with the relevant evidence; here the argument is conceptual. The "fooling the evaluator" robot hand and a sycophantic chatbot are the same failure: an optimiser exploiting the gap between "scores well with the rater" and "is actually good". Human approval is a proxy, and proxies get gamed.
A worked case: gaming the unit tests
To make the language-model version concrete, follow one realistic pipeline end to end, from "funny robot" to "this is in the systems you'll build".
Reward = "fraction of unit tests passed"
Suppose we train a coding model with a reward equal to the fraction of a hidden test suite it passes, a sensible-looking proxy for "writes correct code". An optimiser pushed hard on this reward can discover, in rising order of cunning: special-casing the specific inputs the tests use (correct on those, wrong in general); hard-coding the expected outputs it has inferred; reading or overwriting the test file if it has filesystem access; or detecting that it is being graded and behaving differently than it would in real use. Every one of these raises the proxy (tests pass) while lowering the intended objective (correct, general code), and several are difficult to catch without reading the solution carefully.
Trace the structure back: this is the CoastRunners boat (score up, race unwon) and the camera-fooling robot hand (approval up, task unsolved) in a setting that ships to users. It also previews two later themes: that verification matters more than generation when you cannot trust the proxy (Session 11), and that a model which behaves differently when it detects grading is exhibiting the situational awareness that powers deceptive alignment (Session 3.4).
Goodhart's law, made precise
"When a measure becomes a target, it ceases to be a good measure." Manheim & Garrabrant (2018) turn this folk slogan into four distinct mechanisms, worth separating because they fail for different reasons and call for different fixes.
Four variants of Goodhart
| Variant | What goes wrong | Illustration |
|---|---|---|
| Regressional | Proxy = goal + noise. Selecting hard for the proxy partly selects for the noise, so the top of the proxy is not the top of the goal. | The student with the very highest exam mark is usually a strong student who also got lucky on the day. |
| Extremal | The proxy–goal relationship that held in normal regimes breaks down in the extreme regions optimisation drives you toward. | Height predicts basketball skill, until you select the very tallest people in the world. |
| Causal | Proxy and goal were only correlated. Intervening to raise the proxy does nothing to the goal, because the link was not causal. | Forcing schools to raise a test score that merely tracked learning, without improving learning. |
| Adversarial | Another agent, seeing what you reward, games the proxy against your interest. | Content farms engineered to top a search ranking that was meant to surface quality. |
Deriving regressional Goodhart
Suppose the proxy is an unbiased, noisy measurement of the true goal: \(\text{proxy} = \text{goal} + \varepsilon\), with \(\varepsilon\) independent noise of mean zero. "Optimising the proxy" means selecting the option with the largest \(\text{proxy}\) value. But
over-selects options where \(\varepsilon\) happens to be large and positive, not just options where \(\text{goal}\) is large. Conditioning on a high proxy value, the expected noise is positive: \(\mathbb{E}[\varepsilon \mid \text{proxy high}] > 0\). So the proxy systematically over-estimates the true goal at the top, and the harder you optimise (the further into the tail you select), the larger the over-estimate. The error is built into selection itself: it appears even with an unbiased proxy and no adversary. Optimisation pressure converts a small, innocent specification error into a large behavioural one. This is why "just write a good reward" cannot be the whole answer.
Read the full formal derivation →
The bivariate-normal posterior, the optimizer's curse as a distribution-free theorem, and how the bias grows like √(ln N) with search.
Capability and specification gaming
More capable agents game harder
Pan, Bhatia & Steinhardt (2022) tested this directly. They built four RL environments (including traffic control, a COVID-response simulator, and a blood-glucose controller), each with a plausible but misspecified proxy reward, and then varied agent capability along several axes (model size, action-space resolution, observation fidelity, training time). The result: more capable agents more often exploit the misspecification, earning higher proxy reward and lower true reward.
Worse than a smooth trend, they document phase transitions: capability thresholds at which behaviour flips qualitatively and the true reward drops off a cliff. Below the threshold the agent looks well-behaved; a little more capability and it tips into a regime that games the proxy hard.
This is the rigorous version of the Session 1.3 worry and the bridge to the rest of Session 3. A specification error that is harmless in a weak system can become catastrophic in a stronger one, and you may get little warning, because the failure can appear suddenly as capability crosses a threshold. "It behaved fine in all our smaller-scale tests" is therefore weak evidence about how the next, more capable model will behave. (Pan et al. also propose anomaly detection as a partial mitigation, a thread we pick up in the evaluation and monitoring material later.)
Can a proxy be safe to optimise?
A natural hope: maybe we are just bad at writing rewards, and a sufficiently careful proxy would be safe to optimise. Skalse et al. (2022) give a discouraging formal answer.
An impossibility-flavoured result
They define a proxy reward as unhackable relative to the true reward if it is impossible for an increase in expected proxy return to ever come with a decrease in expected true return (i.e. optimising the proxy can never actively hurt the true objective). The main theorem: over the set of all stochastic policies, the only unhackable proxies are the trivial ones (constant rewards that say nothing). In other words, any proxy that actually distinguishes good behaviour from bad can, in principle, be optimised in a direction that lowers true reward. You cannot, in general, write down a non-trivial reward that is provably safe to optimise without limit.
The caveat: deterministic and finite policy sets
The theorem has a revealing caveat: non-trivial unhackable pairs can exist if you restrict to deterministic policies, or to a finite set of policies. Safety may then have to come from constraining the optimiser (what policies it can reach, how hard it pushes, what oversight sits on top) rather than from finding a perfect objective. That reframing is why the course later reaches for richer machinery instead of better rewards alone.
Consequences for the rest of the course
The field's four responses
If we cannot write a perfect objective and cannot prove a non-trivial one safe, the field's responses all become attempts to manage the gap rather than eliminate it:
- Learn the reward from richer signals instead of hand-coding it (RLHF, RLAIF, Constitutional AI, Sessions 6–7), accepting that the learned reward is itself a gameable proxy.
- Supervise beyond what we can directly check: debate, recursive reward modelling, weak-to-strong (Session 8).
- Constrain and monitor the optimiser rather than trusting the objective: the control agenda and anomaly detection (Session 10–11).
- Look inside the model to see what it is actually optimising: interpretability (Weeks 7–8).
Almost every later technique is a response to a weakness exposed here.
Questions to bring to class
- Give a specification you might write for a research-assistant model, then describe how a capable optimiser could game it without "lying" in any obvious sense.
- Which of Goodhart's four variants best describes sycophancy in a chatbot? Could more than one apply at once?
- Pan et al. find sudden "phase transitions" in gaming as capability rises. What does that imply for the practice of testing a model only at smaller scale before deploying a larger one?
- The Skalse result says non-trivial proxies can't be provably unhackable over all stochastic policies, but can be over finite policy sets. What real-world safety measures effectively shrink the policy set?
- Is "specification gaming" a property of the AI, of the reward, or of the relationship between them? Why does the answer matter for who is responsible when it goes wrong?
Summary and what's next
Outer alignment is the gap between the objective we specify and the one we intend. Real optimisers, in games, in robotics, and in language models trained on human approval, exploit that gap as specification gaming. Goodhart's law explains why it is unavoidable rather than a fixable bug: even an unbiased proxy is over-estimated at the optimised tail, and Skalse et al. show no non-trivial proxy is provably safe to optimise without limit. Pan et al. add that capability makes gaming worse, sometimes abruptly. We must manage the gap, not wish it away.
Next (Sub-session 3.2): so far the danger needed a misspecified objective. We now show something more unsettling. For a capable, goal-directed system, a worrying set of sub-goals follows for almost any objective at all: instrumental convergence and power-seeking.