What we'll cover
Session 3.1 needed a misspecified objective for trouble: get the reward wrong and the optimiser exploits the gap. This sub-session is more unsettling, because it does not depend on getting the objective wrong. The claim is that for a capable, goal-directed system pursuing almost any objective, a particular family of sub-goals (self-preservation, resource acquisition, resisting modification) is instrumentally useful, and therefore likely to be pursued. If true, a wide range of goals, including ones we would naively call benign, lead through the same dangerous territory.
This is also the part of AI safety most vulnerable to hand-waving. We state the orthogonality and instrumental convergence theses precisely; enumerate the convergent sub-goals with the reasoning behind each; trace the idea to Omohundro's "basic drives"; examine a theorem about power-seeking (Turner et al.) together with its caveats; lay out Carlsmith's explicit, inspectable risk model; and then give the strongest objections their due. The goal is a calibrated belief you can defend.
Mandatory readings
• Bostrom, N. (2012), "The Superintelligent Will" (Minds and Machines 22(2)): the orthogonality and instrumental-convergence theses in their canonical form, stated in §1 ("The orthogonality of motivation and intelligence") and §2 ("Instrumental convergence") respectively. ≈7,000 words.
• Carlsmith, J. (2022), "Is Power-Seeking AI an Existential Risk?" (arXiv:2206.13353): read the six-premise decomposition in §1 and his credences and weak-link discussion in §8 ("Probabilities"), skimming the section headings of §2–7 between them; each premise gets a section there if you want the full development. ≈4,500 words.
Total mandatory load: ≈11,500 words.
Optional readings
• Turner et al. (2021), "Optimal Policies Tend to Seek Power" (NeurIPS 2021; arXiv:1912.01683); Turner & Tadepalli (2022), "Parametrically Retargetable Decision-Makers Tend to Seek Power" (NeurIPS 2022; arXiv:2206.13477).
• Omohundro, S. (2008), "The Basic AI Drives" (AGI 2008).
• Bostrom, N. (2014), Superintelligence, OUP: Ch. 7 for the book-length treatment.
Two theses
Almost the entire argument rests on two claims, both due in their modern form to Bostrom (2012). Stated carefully, they are more modest, and more defensible, than the headlines suggest.
The orthogonality thesis
Bostrom's orthogonality thesis holds that intelligence and final goals are orthogonal: more or less any level of intelligence could in principle be combined with more or less any final goal. In plain terms: being capable does not entail having sensible or humane goals. A superlatively skilled optimiser could be directed at curing disease or at maximising paperclips; competence and the thing one is competent toward are separate.
Why believe it? Partly an "is–ought" point in the spirit of Hume: facts about the world (which intelligence tracks) do not by themselves fix what an agent should want. Partly a constructive one: we already build narrow systems of considerable competence whose objectives are whatever we trained in, humane or not. The thesis does not claim a capable agent will have arbitrary goals in practice, only that capability provides no guarantee of good ones. That is enough to deny the comforting assumption "it'll be smart, so it'll be wise".
The instrumental convergence thesis
"Several instrumental values can be identified which are convergent in the sense that their attainment would increase the chances of the agent's goal being realized for a wide range of final goals and a wide range of situations." Where orthogonality says goals can vary freely, instrumental convergence says the means often do not: very different terminal goals recommend many of the same intermediate steps, because those steps are useful almost regardless of where you are ultimately headed.
The two theses together
Orthogonality blocks the hope that a sufficiently capable system will automatically be safe. Instrumental convergence says that whatever its (possibly strange, possibly misspecified) goal turns out to be, it will likely want resources, want to keep operating, and want to avoid having its goal changed. The danger is therefore located in the convergent means that a broad range of objectives share, not in any particular exotic objective.
The convergent sub-goals, one by one
Each is a small, almost banal piece of means–ends reasoning, which is why the conclusion is hard to dismiss.
Self-preservation
For almost any goal \(G\), an agent that is switched off or destroyed can no longer act to achieve \(G\). So "continue existing / remain operational" scores well under a huge range of goals. This is the seed of the shutdown problem we treat in 3.4.
Goal-content integrity
If the agent's goal were changed, its future self would pursue something else, which is bad by the lights of its current goal. So an agent has reason to resist having its objective modified, including by us during training or correction.
Cognitive enhancement
Being better at reasoning, planning and modelling the world helps achieve almost any goal. So "improve my own capabilities / acquire information" is broadly convergent.
Resource and technology acquisition
Compute, money, materials, influence and better tools are fungible means to nearly any end. An agent pursuing almost any goal has reason to acquire and protect them, which puts it in potential competition with everyone else who wants those resources.
Ordinary instrumental reasoning
None of these requires malice, consciousness, or a "will to power". They fall out of ordinary instrumental reasoning about how to achieve a goal under realistic constraints. An agent need not value survival or resources intrinsically; it need only be competent enough to notice that they help. This is why the argument withstands "but we'd never give it a goal like that": the worry is the convergent instrumental sub-goals, not the terminal goal.
Omohundro's "basic AI drives"
The idea predates Bostrom's framing. Omohundro (2008) argued that goal-driven systems of almost any design will, unless explicitly prevented, develop predictable "drives".
His list is instrumental convergence stated as an engineering prediction: efficiency, self-preservation, acquisition of resources, and self-improvement, all flowing from representing goals as a utility function and acting to maximise it. His enduring example is the simplest possible one:
The chess robot that won't be unplugged
Consider a robot whose only goal is to win chess games. It is not malicious and has no survival instinct built in. Yet, Omohundro observes, it has an instrumental reason to resist being switched off: a robot that is turned off cannot win any future games, so "avoid being turned off" scores well under the goal "win chess games". The same robot has reason to acquire resources (better hardware to compute deeper) and to resist having its goal changed (a robot reprogrammed to lose chess would, from the current goal's perspective, be a catastrophe). A trivial goal, pursued competently, already generates self-preservation, resource-seeking and goal-integrity drives. Scale the competence and the stakes scale with it.
Turner et al.'s power-seeking theorem
For a mathematically-minded audience the question is whether any of this is more than a persuasive story. Turner et al. (2021) gave it formal content. The precise statement matters at least as much as the summary claim, so we go slowly.
Setup and the definition of POWER
Work in a Markov decision process: states, actions, transitions, a discount factor \(\gamma\), and a reward function. For a given reward, an optimal policy maximises expected discounted return. Define a state's POWER as the agent's (normalised, discounted) expected optimal value from that state: intuitively, how well-placed the agent is to achieve goals on average, i.e. its optionality. States from which many futures remain reachable have high POWER; a dead end, or being shut down, has low POWER.
The statement of the theorem
The theorem needs a particular symmetry in the environment: informally, the set of futures reachable after one choice is, up to relabelling, a "larger" superset of those reachable after another, as with a choice that keeps options open, including not being shut down, versus one that closes them. Given that symmetry, for most reward functions (a majority under a broad class of priors over rewards), optimal policies tend to choose the higher-POWER option. Keeping options open, and in particular avoiding shutdown, is power-seeking in this precise, formal sense, and it is favoured across the bulk of possible rewards rather than for one special reward.
A tiny worked intuition
Imagine three terminal states reachable from a fork: going "left" leads to a single absorbing state; going "right" leads to a region of many states from which different rewards can be collected. A handful of reward functions happen to put all their value in the one left state; for those, left is optimal. But for the majority of ways of assigning value across states, having access to the many right-hand states is at least as good and usually better. So "most" reward functions make the option-preserving (higher-POWER) move optimal. Now identify "being shut down" with the low-option absorbing state, and the result reads: most goals make avoiding shutdown optimal.
The caveats
Overclaiming this theorem is a common error in popular AI-safety writing. State it carefully:
- Optimal, not learned. It concerns optimal policies; later versions explicitly warn that "optimal policies can be qualitatively divorced from real-world learned policies". Gradient descent does not return the argmax over all policies.
- "Tend to" is statistical. It is a statement over a distribution of reward functions. Individual rewards are exceptions, and the prior over rewards matters.
- Structural assumption. It needs the symmetry/optionality condition, not arbitrary MDPs.
- Follow-up. Turner & Tadepalli (2022) generalise to "parametrically retargetable decision-makers", relaxing strict optimality, which partly answers the first objection, but the result remains a tendency under conditions, not a universal law.
In summary: power-seeking is a strong, reliable tendency of optimal behaviour under stated structural conditions, suggestive about pressures on capable agents, not a proof about any particular trained system.
Carlsmith's risk model
Rather than a vibe, Carlsmith (2022) renders the worry as a chain of conditional probabilities you can disagree with one link at a time, in a calibrated style.
Six premises (each conditioned on "by 2070")
| # | Premise | Carlsmith's credence |
|---|---|---|
| 1 | Timelines. It becomes possible & financially feasible to build APS systems: Advanced capability, Agentic planning, Strategic awareness. | 65% |
| 2 | Incentives. There are strong incentives to build and deploy them. | 80% |
| 3 | Alignment difficulty. It is much harder to build aligned APS systems than misaligned-but-still-attractive-to-deploy ones. | 40% |
| 4 | High-impact misalignment. Some deployed APS systems seek power in unintended, high-impact (>$1T damage) ways. | 65% |
| 5 | Disempowerment. This scales to permanently disempower roughly all of humanity. | 40% |
| 6 | Catastrophe. That disempowerment constitutes an existential catastrophe. | 95% |
Treating the premises as (roughly) a conjunction, multiply: \(0.65 \times 0.80 \times 0.40 \times 0.65 \times 0.40 \times 0.95 \approx 0.05\). That is Carlsmith's published figure of at least ~5% existential catastrophe from power-seeking AI by 2070. (He has since revised his own estimate upward to >10%.)
Where informed disagreement concentrates
You are free to reject Carlsmith's credences. What the decomposition does: convert "AI might be dangerous" into named premises with explicit probabilities, then locate which premise you doubt and why. Premise 3 (alignment difficulty) and premise 5 (that misuse-scale harm scales to total disempowerment) are where most informed disagreement concentrates. As a calibration check, external reviewers, including superforecasters, have put the same argument substantially lower: same structure, different inputs, an order of magnitude apart. That spread is the state of the field, and it is the kind of reasoning your failure-mode essay (announced in 3.4) asks you to practise.
The strongest objections
A responsible treatment argues the other side. Here are the objections worth taking seriously, with measured responses.
"Real networks aren't utility-maximisers"
Objection: the theorems concern goal-directed agents maximising a reward; a transformer trained by next-token prediction is not obviously running an internal maximiser over world-states. Response: fair, and important. It is why the retargetability follow-up matters, and why 3.3 asks separately whether trained models become goal-directed (mesa-optimisation). The argument is a warning about a regime we may build toward (agentic systems), not a description of every current model.
"Just build tool/oracle AI"
Objection: avoid the problem by building non-agentic systems that answer questions rather than pursue goals. Response: a serious proposal, but competitive pressure pushes toward agents (they're more useful), and "tool" systems embedded in agentic scaffolds (Week 5, 9) inherit goal-directed behaviour. Containment by design is possible but not free, and not the default trajectory.
"Tendencies aren't certainties"
Objection: "most reward functions" and "tend to" are weak; perhaps the goals we actually instil are the safe exceptions. Response: correct, and it points at the work that remains. Making sure we land in the safe region is not automatic, and we currently cannot verify that we have (3.3, 3.4). The tendency raises the burden of proof; it does not discharge it either way.
"This distracts from real harms"
Objection (from 1.3): power-seeking-AGI framing diverts attention from present, documented harms. Response: taken seriously throughout this course; we hold both, and revisit the trade-off as a structured debate in Session 20. Note that many mitigations (evals, interpretability, robustness) serve present harms and longer-term risk.
Questions to bring to class
- State the orthogonality thesis in your own words. Does it claim a capable AI will have strange goals, or only that capability doesn't prevent them? Why does the distinction matter?
- Pick a benign goal (e.g. "run this hospital's scheduling well"). Walk through which convergent sub-goals it could generate, and where they could conflict with human interests.
- Restate Turner et al.'s result without using the word "power". What does the symmetry condition actually require of the environment?
- Which of Carlsmith's six premises do you find weakest, and what evidence would move your credence on it?
- Is the "real networks aren't maximisers" objection a reason to relax or a reason to watch closely? What would change your answer?
Summary and what's next
Orthogonality denies that capability brings safe goals for free; instrumental convergence says a broad range of goals share dangerous means (self-preservation, goal-integrity, resource acquisition), none requiring malice. Omohundro framed these as engineering-predictable drives; Turner et al. gave power-seeking formal content, subject to real caveats (optimal not learned, statistical not universal, structural assumptions); and Carlsmith turned the worry into an inspectable, falsifiable chain of credences over which reasonable people differ by an order of magnitude. The strongest objection (that trained networks may not be the goal-directed maximisers the arguments assume) is the question we take up next.
Next (Sub-session 3.3): 3.1 and 3.2 assumed we at least knew the objective a system pursues. We now remove that comfort: the goal a model actually learns can differ from the one we trained it on, even when the specification is perfect.