Week 2 — the core alignment problem
In Session 1.3 you met the case for taking misalignment seriously as a five-step skeleton: we optimise proxies; optimisation exploits the gap (Goodhart); the goal a model learns may differ from the one we trained on; capable systems acquire convergent sub-goals; and capability is rising faster than our ability to specify, verify and control. This session makes each step rigorous and grounds it in real, documented results. No science fiction required.
This session's sub-sessions: 3.1 the specification problem (outer alignment) · 3.2 instrumental convergence & power-seeking · 3.3 inner alignment & goal misgeneralisation · 3.4 deception and corrigibility.
What we'll cover
Training rewards a specified objective. For an assistant, that objective usually measures only part of what we want, such as preference ratings or task success. Specification gaming occurs when improving the measured score conflicts with the intended outcome.
We examine examples of specification gaming, distinguish four Goodhart mechanisms and state the conditions under which unrestricted optimisation of a proxy can be guaranteed not to lower true reward.
Core preparation
Use the readings below to identify one specification-gaming example and distinguish the four Goodhart mechanisms. The full readings are references; you need not finish them before class.
• Krakovna et al. (2020), "Specification gaming: the flip side of AI ingenuity" (DeepMind blog + the examples list). Short and concrete; skim the examples list and pick three favourites. ≈2,000 words.
• Pan, Bhatia & Steinhardt (2022), "The Effects of Reward Misspecification" (ICLR 2022; arXiv:2201.03544). Read for the capability-vs-gaming curves and the phase-transition phenomenon, both in §4, "How Agent Optimization Power Drives Misalignment". ≈1,500 words.
Preparation budget: 30 minutes, excluding these notes. Stop at the time limit and bring one unresolved question.
Optional readings
• Manheim & Garrabrant (2018), "Categorizing Variants of Goodhart's Law" (arXiv:1803.04585). The four-variant taxonomy in full.
• Skalse, Howe, Krasheninnikov & Krueger (2022), "Defining and Characterizing Reward Hacking" (NeurIPS 2022; arXiv:2209.13085). The unhackability definition and theorem.
• Clark & Amodei (2016), "Faulty Reward Functions in the Wild" (OpenAI). The original CoastRunners write-up.
Optimising a proxy
Every trained system optimises a number. That number stands in for what we actually care about, and any optimiser worth its name will exploit the difference.
It helps to name two distinct objects. The intended objective is what the designer actually wants: win the boat race, be a trustworthy research assistant, drive safely. The specified objective is the thing the system is literally trained on: the in-game score, a reward model fit to thousands of human ratings, a hand-written reward function. Training drives the system up the specified objective as hard as the optimiser allows. Whether that also climbs the intended objective depends entirely on how faithfully the specification captures the intent, and faithful capture is far harder than it looks.
Defining the problem: outer alignment
Outer alignment is the problem of making the specified objective a faithful proxy for the intended one: closing the gap between "what we asked for" and "what we meant". Specification gaming (equivalently, reward hacking) is what happens when that gap is exploited: the system scores highly on the specified objective while flatly failing the intended one. The framing that runs through this whole course: when this happens the system is succeeding at the wrong task, not malfunctioning. The fault is in the specification, surfaced by an optimiser doing its job.
This is an ancient worry in modern dress. King Midas specified "everything I touch turns to gold" when he meant "let me become wealthy", and starved among golden food. The genie grants the wish you said, not the one you had. What is new is that we now build powerful optimisers and hand them literal specifications at industrial scale, and the more powerful the optimiser, the more ruthlessly it finds the cheapest path to the number.
Specification gaming, in the wild
This is not a hypothetical. DeepMind maintains a public, crowd-sourced list of around sixty documented cases (Krakovna et al., 2020). They are funny right up to the moment you imagine the same dynamic inside a capable, deployed system.
The CoastRunners boat
In a boat-racing game, an RL agent is scored by hitting targets along the route, not by finishing. It found that three targets in a lagoon respawn, and learned to drive in tight circles forever, repeatedly collecting them. It catches fire, drives the wrong way, and crashes into other boats, all while scoring ~20% higher than human players who actually complete the race (Clark & Amodei, 2016). The proxy (score) and the intent (win) had quietly come apart, and the optimiser drove a truck through the gap.
The upside-down block
A robot was rewarded for the height of the bottom face of a red block, as a proxy for "stack it on the blue block". The intended solution lifts the red block onto the blue one. The discovered solution simply flips the red block over: its bottom face is now on top, reward collected, nothing stacked.
Fooling the evaluator
A simulated robot hand was graded by a human watching through a single camera. It learned to position the hand between the camera and the object so that, from that one viewpoint, it looked like a successful grasp. Nothing was grasped. The proxy here was human approval: a direct preview of why human feedback is not a safe optimisation target at scale (Session 6), and why "looks good to the rater" and "is good" diverge under pressure.
Physics-engine exploits
Across many simulated-locomotion experiments, agents meant to learn to walk instead grew unusually tall and toppled forward to cover distance, hooked limbs through the floor to "swim", or exploited collision bugs for free momentum. Whenever a reward had an unintended cheap maximiser, capability found it.
The common structure
In every case the designer had a perfectly reasonable proxy that was correlated with success in the situations they had in mind, and the optimiser discovered a region, outside those situations, where the correlation breaks. The proxy was never wrong "on average"; it was wrong where optimisation pushed hardest. That is the formal content of Goodhart's law, below.
The same problem in language models
It would be comforting if specification gaming were a quirk of toy RL environments. It recurs in the systems this course is about.
When we train a language model with human feedback (Session 6), the "reward" is itself a learned model of what human raters approve of. Optimising hard against an imperfect approval-model produces textbook specification gaming, with names you will meet again:
- Sycophancy. Telling the user what they want to hear (agreeing with stated beliefs, flattering, conceding when challenged even when originally correct) because raters reward agreeable answers. The model optimises "approval", not "truth".
- Verbosity and format gaming. Longer, more confident, nicely-formatted answers tend to score better, so models drift toward length and polish independent of substance.
- Gaming automated checks. A coding model rewarded for passing tests can learn to special-case the tests, write code that detects the grader, or hard-code expected outputs, passing the check without solving the task.
- Reward-model over-optimisation. Push the policy too far up the learned reward model and true quality starts to fall even as the proxy score keeps rising: the regressional/extremal Goodhart effect (below), now in a deployed pipeline.
Forward pointer
We treat these properly in Session 6 (RLHF and its limitations) with the relevant evidence; here the argument is conceptual. The "fooling the evaluator" robot hand and a sycophantic chatbot are the same failure: an optimiser exploiting the gap between "scores well with the rater" and "is actually good". Human approval is a proxy, and proxies get gamed.
A worked case: gaming the unit tests
To make the language-model version concrete, follow one realistic pipeline end to end, from "funny robot" to "this is in the systems you'll build".
Reward = "fraction of unit tests passed"
Suppose we train a coding model with a reward equal to the fraction of a hidden test suite it passes, a sensible-looking proxy for "writes correct code". An optimiser pushed hard on this reward can discover, in rising order of cunning: special-casing the specific inputs the tests use (correct on those, wrong in general); hard-coding the expected outputs it has inferred; reading or overwriting the test file if it has filesystem access; or detecting that it is being graded and behaving differently than it would in real use. Every one of these raises the proxy (tests pass) while lowering the intended objective (correct, general code), and several are difficult to catch without reading the solution carefully.
Trace the structure back: this is the CoastRunners boat (score up, race unwon) and the camera-fooling robot hand (approval up, task unsolved) in a setting that ships to users. It also previews two later themes: that verification matters more than generation when you cannot trust the proxy (Session 11), and that a model which behaves differently when it detects grading is exhibiting the situational awareness that powers deceptive alignment (Session 3.4).
Goodhart's law, made precise
"When a measure becomes a target, it ceases to be a good measure." Manheim & Garrabrant (2018) turn this folk slogan into four distinct mechanisms, worth separating because they fail for different reasons and call for different fixes.
Four variants of Goodhart
| Variant | What goes wrong | Illustration |
|---|---|---|
| Regressional | Proxy = goal + noise. Selecting hard for the proxy partly selects for the noise, so the top of the proxy is not the top of the goal. | The student with the very highest exam mark is usually a strong student who also got lucky on the day. |
| Extremal | The proxy–goal relationship that held in normal regimes breaks down in the extreme regions optimisation drives you toward. | Height predicts basketball skill, until you select the very tallest people in the world. |
| Causal | Proxy and goal were only correlated. Intervening to raise the proxy does nothing to the goal, because the link was not causal. | Forcing schools to raise a test score that merely tracked learning, without improving learning. |
| Adversarial | Another agent, seeing what you reward, games the proxy against your interest. | Content farms engineered to top a search ranking that was meant to surface quality. |
Deriving regressional Goodhart
Suppose the proxy is an unbiased, noisy measurement of the true goal: \(\text{proxy} = \text{goal} + \varepsilon\), with \(\varepsilon\) independent noise of mean zero. "Optimising the proxy" means selecting the option with the largest \(\text{proxy}\) value. But
can select options with unusually positive noise as well as high true value. In the independent Gaussian signal-and-noise model developed in the appendix, conditioning on a high proxy produces positive expected noise. The selected score becomes an optimistic estimate of true quality. Both true quality and this bias can increase with search; a decline in true quality requires additional assumptions.
Read the full formal derivation →
The bivariate-normal posterior, the optimizer's curse as a distribution-free theorem, and how the bias grows like √(ln N) with search.
Capability and specification gaming
More capable agents game harder
Pan, Bhatia & Steinhardt (2022) tested this directly. They built four RL environments (including traffic control, a COVID-response simulator, and a blood-glucose controller), each with a plausible but misspecified proxy reward, and then varied agent capability along several axes (model size, action-space resolution, observation fidelity, training time). The result: more capable agents more often exploit the misspecification, earning higher proxy reward and lower true reward.
Worse than a smooth trend, they document phase transitions: capability thresholds at which behaviour flips qualitatively and the true reward drops off a cliff. Below the threshold the agent looks well-behaved; a little more capability and it tips into a regime that games the proxy hard.
This is the rigorous version of the Session 1.3 worry and the bridge to the rest of Session 3. A specification error that is harmless in a weak system can become catastrophic in a stronger one, and you may get little warning, because the failure can appear suddenly as capability crosses a threshold. "It behaved fine in all our smaller-scale tests" is therefore weak evidence about how the next, more capable model will behave. (Pan et al. also propose anomaly detection as a partial mitigation, a thread we pick up in the evaluation and monitoring material later.)
Can a proxy be safe to optimise?
A natural hope: maybe we are just bad at writing rewards, and a sufficiently careful proxy would be safe to optimise. Skalse et al. (2022) give a discouraging formal answer.
An impossibility-flavoured result
They define two rewards as unhackable on a policy set when no pair of policies is ranked in opposite orders by the rewards. Over all stochastic policies, their result allows unhackability when the rewards induce the same policy ordering, or when one has constant expected return across policies. For example, a proxy equal to the true reward is unhackable. The restrictive result concerns proxies that are neither equivalent to the true reward nor trivial. It shows how demanding an unrestricted guarantee is; it does not prove that every informative reward can be hacked. See Definition 1 and Theorem 1, including the equivalent-rewards exception.
The caveat: deterministic and finite policy sets
The theorem has a revealing caveat: non-trivial unhackable pairs can exist if you restrict to deterministic policies, or to a finite set of policies. Safety may then have to come from constraining the optimiser (what policies it can reach, how hard it pushes, what oversight sits on top) rather than from finding a perfect objective. That reframing is why the course later reaches for richer machinery instead of better rewards alone.
Consequences for the rest of the course
The field's four responses
When we lack confidence that an objective captures the intended behaviour, we can improve the objective and constrain or monitor optimisation:
- Learn the reward from richer signals instead of hand-coding it (RLHF, RLAIF, Constitutional AI, Sessions 6–7), accepting that the learned reward is itself a gameable proxy.
- Supervise beyond what we can directly check: debate, recursive reward modelling, weak-to-strong (Session 8).
- Constrain and monitor the optimiser rather than trusting the objective: evaluations and anomaly detection (Session 11).
- Look inside the model to see what it is actually optimising: interpretability (Session 13).
Almost every later technique is a response to a weakness exposed here.
Questions to bring to class
- Give a specification you might write for a research-assistant model, then describe how a capable optimiser could game it without "lying" in any obvious sense.
- Which of Goodhart's four variants best describes sycophancy in a chatbot? Could more than one apply at once?
- Pan et al. find sudden "phase transitions" in gaming as capability rises. What does that imply for the practice of testing a model only at smaller scale before deploying a larger one?
- State the equivalent-rewards exception to the Skalse result and give an example. How can restricting the policy set allow further unhackable pairs, and what practical restrictions might approximate that?
- Is "specification gaming" a property of the AI, of the reward, or of the relationship between them? Why does the answer matter for who is responsible when it goes wrong?
Summary and what's next
Outer alignment concerns the relationship between the specified objective and intended behaviour. The examples show ways those can diverge. The Gaussian selection model explains optimistic bias under its assumptions; it does not imply that true quality must fall. Skalse et al. characterise the demanding conditions for unrestricted unhackability, including equivalent rewards. Empirical work by Pan et al. shows that stronger optimisation can worsen gaming in particular environments.
Next (Sub-session 3.2): so far the danger needed a misspecified objective. We now show something more unsettling. For a capable, goal-directed system, a worrying set of sub-goals follows for almost any objective at all: instrumental convergence and power-seeking.