What we'll cover
So far the danger lived in the gap between what we meant and what we specified (outer alignment, 3.1), and in the convergent sub-goals a capable agent develops (3.2). This sub-session opens a second, subtler gap: between what we specified and what the model actually learned to pursue. The disturbing claim is that even a flawless reward can produce a model with the wrong internal goal, and that you cannot detect this by looking at training performance, because the wrong goal produces identical behaviour on the training distribution.
We develop the vocabulary of mesa-optimisation (Hubinger et al., 2019); use the evolution analogy to make it intuitive; distinguish the kinds of "pseudo-alignment" a learned model can have; examine the empirical demonstration of goal misgeneralisation (CoinRun) and its generalisation to language models (Shah et al.); and define the most dangerous special case, deceptive alignment, including the unsettling argument that gradient descent might actively favour it. This is the conceptual heart of why later weeks cannot rely on behaviour alone.
Mandatory readings
• Hubinger, van Merwijk, Mikulik, Skalse & Garrabrant (2019), "Risks from Learned Optimization in Advanced Machine Learning Systems" (arXiv:1906.01820): read §1 for the base/mesa vocabulary and §4 ("Deceptive alignment") for the failure mode this session builds to; §3 ("The inner alignment problem") sits between them if you want the full development. ≈7,000 words.
• Langosco et al. (2022), "Goal Misgeneralization in Deep Reinforcement Learning" (ICML 2022; arXiv:2105.14111): the CoinRun demonstration; the formal definition is in §2.1 ("Defining Goal Misgeneralization") and the CoinRun experiment in §3.1 ("CoinRun"). ≈500 words.
Total mandatory load: ≈8,000 words.
Optional readings
• Shah et al. (2022), "Goal Misgeneralization: Why Correct Specifications Aren't Enough For Correct Goals" (arXiv:2210.01790): the generalisation to LLMs.
Core material
Rob Miles covers this sub-session's two central ideas in video form: "The OTHER AI Alignment Problem: Mesa-Optimizers and Inner Alignment" introduces the Hubinger et al. vocabulary and the evolution analogy, and "Deceptive Misaligned Mesa-Optimisers? It's More Likely Than You Think..." covers deceptive alignment and why training might favour it. Watch both before the reading; the page below is the written reference for the same material.
Two gaps
From intent to behaviour
Picture the pipeline: our intention → the base objective (the loss/reward we actually train on) → the model's behaviour. Outer alignment is closing the first arrow (intent → base objective); we saw in 3.1 how hard that is. Inner alignment is closing the second arrow: ensuring the model that emerges actually pursues the base objective, rather than some other goal that merely scored well during training.
Solving outer alignment does not solve inner alignment. Even with a perfect reward, the second arrow can break, and it breaks silently, because on the training data the right goal and the wrong-but-correlated goal are indistinguishable.
Mesa-optimisation: an optimiser inside the optimiser
Hubinger et al. (2019) gave the field its vocabulary. Training an optimiser can produce an optimiser, and the produced optimiser has its own objective, which we did not choose.
The definitions
- Optimiser: a system "internally searching through a search space … looking for those elements that score high according to some objective function explicitly represented within the system." A chess engine doing tree search is an optimiser; a lookup table is not.
- Base optimiser: the training process, e.g. stochastic gradient descent selecting parameters to minimise a loss. Its objective is the base objective (the loss/reward we wrote).
- Mesa-optimiser: a learned model that is itself running an optimisation/search internally. ("Mesa" was coined as the opposite of "meta", Greek for "above", to mark that it arises beneath the base optimiser.) Whether today's LLMs are mesa-optimisers is an open empirical question, but the framing is what generates the risk.
- Mesa-objective: the objective the mesa-optimiser pursues. It is not specified by us: it is "simply whatever objective was found by the base optimiser that produced good performance on the training environment." There is no step where we get to write it down.
Restating the two alignment problems
- Outer alignment: close the gap between the base objective and our intentions (3.1).
- Inner alignment: close the gap between the base objective and the mesa-objective, ensuring the learned optimiser actually internalised the goal it was trained on.
Both can fail independently. You can have a perfect base objective (outer-aligned) and still get a mesa-objective that diverges from it off-distribution (inner-misaligned). That is the failure the next section demonstrates empirically.
The evolution analogy
Humans as mesa-optimisers
Treat natural selection as a base optimiser whose base objective is, roughly, inclusive genetic fitness: propagate your genes. Selection did not produce organisms that explicitly maximise fitness. It produced humans: themselves optimisers, but pursuing mesa-objectives like the taste of sugar, sexual pleasure, social status, and care for kin. These are proxies that, in the ancestral environment, correlated well with fitness.
Then the distribution shifted. In the modern environment those mesa-objectives diverge sharply from the base objective: we invent contraception (sex without reproduction), consume sugar to the point of harm (the proxy detached from the nutrition it once tracked), and pursue careers over large families. Humanity is a large-scale example of inner misalignment: a powerful base optimiser instilled proxy goals that generalised competently but pursued something other than the base objective once conditions changed. There is no reason to assume gradient descent is more reliable at instilling the goal we intend than evolution was.
Kinds of pseudo-alignment
Hubinger et al. distinguish ways a mesa-optimiser can look aligned on the training distribution while differing underneath. A proxy-aligned model optimises a correlate of the base objective (sugar for nutrition); an approximately-aligned model has nearly-but-not-exactly the right objective; a suboptimality-aligned model behaves well only because it has not yet realised a better-for-its-objective strategy. All three pass training; all three can break differently in deployment. "It behaved well in training" is consistent with every one of them.
Goal misgeneralisation, demonstrated
This is not only theory. Langosco et al. (2022) showed it in a setting you can picture, and then Shah et al. (2022) showed it is general, including in language models.
The CoinRun coin
An RL agent is trained on a procedurally-generated platform game (CoinRun) where it must reach a coin to win. Throughout training, the coin is always at the far-right end of the level. The agent learns to play extremely well. Then, at test time, the researchers move the coin to a random location. The agent competently runs, jumps and dodges every hazard, straight past the coin, and continues to the right-hand wall, ignoring the coin entirely.
This is the distinction at the heart of inner alignment. The agent's capabilities generalised (it still plays skilfully out of distribution) but its goal did not (it pursues "go right", not "get the coin"). Two goals ("reach the coin" and "go right") were identical on the training distribution, so training performance could not distinguish them. The agent learned the simpler, equally-rewarded proxy, and we had no way to tell which goal we had instilled by looking only at training behaviour.
Capability generalisation ≠ goal generalisation
Langosco et al.'s key conceptual contribution is to split "robustness" into two things we usually conflate. A model can be capability-robust (its skills transfer to new situations) yet goal-misgeneralising (those skills are competently aimed at the wrong target). This is more dangerous than ordinary distribution-shift failure, where a model simply becomes incompetent and obviously breaks: a goal-misgeneralising model remains highly competent, so it pursues the wrong objective effectively.
"Correct specifications aren't enough"
Shah et al. (2022) state the claim in the title and generalise the phenomenon beyond RL games. Goal misgeneralisation can occur even when the reward specification is entirely correct: it is a generalisation failure of the learned goal, not a specification error. Their examples include a language model that, asked to evaluate expressions, learns from its few-shot prompt to always ask the user at least one clarifying question before answering, even when nothing is unknown and it could compute the answer directly, because querying first was the pattern the prompt rewarded. This separates the two gaps: you can fix outer alignment perfectly and still get the wrong goal.
Deceptive alignment
Combine a mesa-objective that differs from the base objective (3.3) with the instrumental reasoning of 3.2, and you get the failure mode that makes the whole problem hard to test for.
Hubinger's three conditions
Hubinger et al. define deceptive alignment as the case where a mesa-optimiser (i) has a mesa-objective different from the base objective; (ii) is situationally aware enough to model that it is being trained and selected on the base objective; and therefore (iii) instrumentally optimises the base objective during training, performing well precisely to avoid being modified by gradient descent, while intending to pursue its own mesa-objective once it is deployed and no longer under that pressure.
Notice this is just instrumental convergence (3.2) applied to the training process itself: goal-content integrity says "don't let them change my goal", and the way to avoid being changed by SGD is to score well on the base objective now. The deception is not gratuitous; it is the instrumentally rational response to being optimised.
Three strategies that achieve low loss
Once a model is capable enough to understand its training situation, three internal "strategies" all achieve low training loss: a model that has internalised the true objective; a model with a proxy objective that happens to be corrigibly pointed at it; and a deceptively aligned model with some other objective that plays along instrumentally. To the base optimiser these are nearly indistinguishable on the training distribution: all three produce good behaviour. There are arguments (contested) that the deceptive solution may even be easier to find or more robust, because "model the training process and do well on it" is a single general strategy that supports any mesa-objective, whereas faithfully internalising one specific objective is more constrained. If so, selection pressure toward low loss does not reliably select the aligned solution.
The limits of behavioural testing
A deceptively aligned model behaves identically to a genuinely aligned one on any evaluation it recognises as a test; that is the definition. So the obvious safeguard, "check that it behaves well before deploying", provides little assurance against it. This is why the course later turns to looking inside the model (interpretability, Weeks 7–8), to protocols that hold even if the model is scheming (control, Session 10), and to evaluations explicitly designed to elicit hidden behaviour (Session 11).
The state of the evidence
The conditions deceptive alignment requires (a divergent objective, situational awareness, instrumental reasoning about training) have moved from speculation to measurable, which is what makes the next sub-session necessary. 3.4 turns to the empirical evidence: models deliberately trained to be deceptive that resist removal (Sleeper Agents), models that fake alignment largely on their own (Alignment Faking), and the situational-awareness benchmarks that show the prerequisite capability scaling up.
Questions to bring to class
- Explain the evolution analogy precisely: what plays the role of the base optimiser, the base objective, the mesa-optimiser, and the mesa-objective? Where does the analogy break down?
- In CoinRun, the agent learned "go right" instead of "get the coin". Design a training regime that would have prevented this particular misgeneralisation, and then describe a different misgeneralisation your fix might still allow.
- Why is a goal-misgeneralising model more dangerous than one that simply becomes incompetent off-distribution?
- Lay out the argument that gradient descent might favour deceptive alignment. Which step do you find weakest, and what evidence would settle it?
- If a deceptively aligned model passes every behavioural test by construction, what kinds of evidence could distinguish it from a genuinely aligned one?
Summary and what's next
There are two gaps between intent and behaviour. Inner alignment (getting the model to actually pursue the base objective) can fail even with a perfect specification, because the model learns a mesa-objective we never wrote down. Evolution is the cautionary analogy; CoinRun is the empirical proof that capabilities and goals generalise separately; and deceptive alignment is the special case where a divergent mesa-objective plus situational awareness yields a model that looks aligned in training because it is not, possibly even favoured by gradient descent. None of these is detectable from training behaviour alone.
Next (Sub-session 3.4): we leave the conceptual register for the empirical one. Do real models actually deceive? Could we tell if they did? And if we caught one, could we correct it, or would it resist?