What we'll cover
A course that only made the case for its own subject would be advocacy. So this sub-session does both, and takes the second part as seriously as the first. We lay out the strongest version of the argument that misalignment deserves serious technical work, and the four background premises beneath it. Then the strongest arguments against: capabilities scepticism, the "your arguments are too informal" objection, the present-harms critique (which has a real Global South dimension), and the ideological critique. We ask whether the supposed trade-off between present and long-term harms is even real, treating that as an empirical question. And we set out the stance this course takes: hold each claim at the confidence the evidence supports.
Mandatory readings
• Hanna, A. & Bender, E. M. (2023), "AI Causes Real Harm. Let's Focus on That over the End-of-Humanity Hype" (Scientific American): the present-harms case, stated briskly. ≈1,500 words.
• Soares, N. (2015), "Four Background Claims" (intelligence.org): the field's premises, each stated with its standard objection and response; the basis for the four claims below. ≈2,000 words.
• Narayanan, A. & Kapoor, S. (2025), "AI as Normal Technology" (Knight First Amendment Institute): the strongest systematic statement of capabilities scepticism. Read three specific parts: the untitled opening (the framework in one pass), then two sections from Part II: "Human ability is not constrained by biology" (the intelligence-versus-power reframe, aimed at background claims 2 and 3 below) and "Games provide misleading intuitions about the possibility of superintelligence" (why chess-derived intuitions mislead, aimed at claim 2). The full essay (~25,000 words) is optional below, and we return to it in Session 20. ≈3,000 words.
Total mandatory load: ≈6,500 words.
Optional readings
• Hendrycks, D. (2024), Intro to ML Safety — "X-Risk Overview" (or AISES textbook Ch. 1; arXiv:2411.01042): the full book-chapter version of the risk taxonomy assigned in 1.2; read it if you want the long-term-risk case at length. ≈24,000 words (full book chapter).
• Tegmark, M. (2017), Life 3.0, ch. 6 (Penguin Random House): background for the second claim's outer bound; the passage on the physical limits of computation is the relevant part. ≈13,000 words (full book chapter, estimate).
• Bender, Gebru, McMillan-Major & Shmitchell (2021), "On the Dangers of Stochastic Parrots" (FAccT 2021, pp. 610–623; DOI 10.1145/3442188.3445922): critics and counterpoint; the stochastic-parrots argument.
• Gebru, T. & Torres, É. P. (2024), "The TESCREAL bundle…" (First Monday 29(4); DOI 10.5210/fm.v29i4.13636): the ideological critique.
• Hoes, E. & Gilardi, F. (2025), "Existential risk narratives about AI do not distract from its immediate harms" (PNAS 122(16)): the empirical counterpoint.
• Narayanan & Kapoor, "AI as Normal Technology", the full essay (Knight First Amendment Institute): beyond the mandatory sections, Part I argues that diffusion speed, not capability, sets the pace (its "Benchmarks do not measure real-world utility" section pairs with Sessions 2.3–2.4), and Parts III–IV (risks, policy) connect to Sessions 12 and 20.
The five claims
Here is the argument as a chain of claims, each of which you can inspect and attack. The rigorous version is Session 3. This is the skeleton, so you can see where the joints are.
- We optimise proxies, not intentions. We cannot write "be helpful, honest and harmless" as a loss function, so we optimise measurable stand-ins: human preference, a reward model. The target is never exactly what we meant (Session 3.1).
- Optimisation exploits the gap. Push hard on a proxy and the system finds the cheapest way to score, which need not be the thing you wanted. This is Goodhart's law, and it is provable, not just observed (the derivation appendix to 3.1).
- The goal a model learns may differ from the one we trained on. Even a perfect proxy can yield a model with an unintended internal objective that only coincidentally matched during training. This is inner misalignment, demonstrated empirically (Session 3.3).
- Capable systems acquire convergent sub-goals. For almost any objective, staying operational, keeping options open and acquiring resources all help. So capable optimisers tend toward self-preservation and influence (Session 3.2).
- Capability is rising fast and predictably; safety is not. Scaling laws make capabilities forecastable (Session 2.3). We have no comparable handle on "won't deceive its overseer". The worry is the gap and its trajectory, not any single model today.
The conjunction of the claims
To get from these claims to "existential catastrophe is likely", you must conjoin several of them, plus extras like "this scales to the whole world" and "we won't fix it in time". Conjunction makes the strong conclusion fragile. Doubt in any one link multiplies through, which is why careful estimates of catastrophic risk vary by orders of magnitude (we see Carlsmith's explicit version in Session 3.2). But conjunction also means the weaker conclusion, that this is a serious problem worth technical work, needs far less. Even modest credence in the early links, given the stakes, is enough to justify the field. Most of this course lives at that weaker, sturdier conclusion.
Four background claims
Beneath the five-claim chain sit older, more basic premises. Soares (2015) states four of them, each with the standard objection and his response, so both sides of the argument arrive already structured. Much disagreement about the field turns out to be disagreement about one of these.
1 · Humans have general intelligence
The claim: humans have a versatile ability to solve problems across many domains.
The objection: there is no such general faculty. Humans run a collection of special-purpose modules, and nothing correspondingly general will transfer to machines.
The response: whatever produces it, human cognition succeeds in domains evolution never prepared us for, such as spacecraft, vaccines and computers. That cross-domain reach is what matters, and a machine that matched it would have the same reach.
2 · AI could become far more intelligent than humans
The claim: AI systems could substantially exceed human intelligence.
The objection: brains may have properties no machine can replicate; the algorithms may be too complex to build; or humans may already sit near the ceiling of possible intelligence.
The response: brains are physical systems, and no known physics privileges biological computation. Evolution worked under tight constraints (energy budgets, skull size, neuron signalling far slower than electronics) that engineered systems do not share, and machines gain speed, copying and editing advantages that biology cannot.
3 · Highly intelligent AI would shape the future
The claim: systems far more capable than us would substantially determine what happens next.
The objection: however capable, such systems would still have to work with us and integrate into our economy rather than steer around it.
The response: intelligence is exactly what let humans, rather than chimpanzees, decide how the planet gets used. Integration with our economy is a strategy, not a law of nature. A sufficiently capable system could invent alternatives, persuade, or act faster than our institutions respond.
4 · Beneficial outcomes require deliberate design
The claim: good outcomes from powerful AI are not the default. They must be engineered.
The objection: smarter systems will be better at working out what we value, and better at acting on it.
The response: knowing what we value is not the same as caring about it. A system optimises whatever objective it actually has, and capability does not pull that objective toward benevolence. A perfect model of human values, attached to the wrong goal, just predicts us better while pursuing something else.
Read the post before class and arrive with a position. Which claim carries your lowest credence? And does the paired objection actually target it, or does the response defuse it? On the second claim, the outer bound is physical rather than biological. Tegmark (Life 3.0, ch. 6) works through the limits physics places on computation and finds them astronomically far above current hardware, so "humans are near the ceiling" must be argued on other grounds. Bostrom makes the hardware-advantage version of the same point in Superintelligence, ch. 3. These are discussion points, not settled premises; the in-class exercise below and the Session 20 seminar both draw on them.
The strongest critics
Take these seriously. The best critics are not careless, and many are leading researchers. They are usually disputing which risks deserve attention and resources, not denying that AI can do harm.
"Focus on present harms"
Hanna & Bender, and the DAIR group founded by Timnit Gebru, argue that speculative-extinction talk diverts attention, funding and regulatory energy from documented harms happening now. Discrimination, exploited data-labour, surveillance and environmental cost all fall hardest on the Global South. On this view the very framing of "x-risk" is a political act with distributional consequences.
Capabilities scepticism
Perhaps today's systems are less capable than the hype implies: "stochastic parrots" (Bender et al., 2021) that recombine training text without understanding, and will fail in ways that don't scale to agency. If so, the dangerous-agency story is premature, and apparent "emergence" may be partly a measurement artefact. We test this directly in Session 2.4.
"The arguments are too informal"
A methodological objection. The core arguments lean on notions that don't cleanly apply to a network trained by gradient descent on next-token prediction: a "goal", an "optimiser", "wanting". Without rigour, the conclusions are unearned. This is a fair challenge, and one reason this course insists on formal and empirical grounding rather than intuition.
The ideological critique
Gebru & Torres (2024) argue that the long-termist milieu carries a specific bundle of ideologies (their "TESCREAL" acronym) with troubling intellectual roots, and that this shapes which questions get asked and funded. Whatever you make of it, it is a serious prompt to ask who set this field's agenda and whose concerns it centres. This course's African framing presses the same question.
Whether the trade-off is real
Present harms versus long-term risk: the evidence
Much of the heat assumes the two concerns compete for a fixed pool of attention. But that is an empirical claim, and it can be tested. Hoes & Gilardi (PNAS, 2025) ran experiments on whether exposure to existential-risk narratives reduces people's concern for AI's immediate harms. It does not measurably crowd them out. One study doesn't settle the funding-and-power argument, which is about institutions rather than individual attitudes. But it should make you cautious about treating the trade-off as obvious in either direction.
Note too that many technical mitigations serve both concerns. Better evaluations catch present misuse and dangerous capabilities. Interpretability exposes bias and deception. Robustness work helps against today's jailbreaks and tomorrow's. Framing this as a zero-sum fight obscures how much of the actual work is shared.
The stance this course takes
Three commitments
You do not have to believe in catastrophe to find this material worth doing, and you do not have to dismiss long-term risk to take present harms seriously. The position is to (1) hold each claim at the confidence the evidence supports, as a probability rather than a verdict; (2) be able to state the other side's strongest argument before giving your own; and (3) update when the evidence does. We treat "this premise is uncertain" as a finding. That habit is assessed: both the failure-mode essay and the Session 20 sceptics seminar reward steelmanning over conclusion-picking.
Making a vague claim precise
When you find yourself agreeing or disagreeing with "AI is dangerous", force the vague claim into structure. Which risk type (1.2)? Which link in the chain above? What credence, and what evidence would move it? "I put maybe 20% on premise 3 because the empirical case for inner misalignment in LLMs is still thin" is a position you can defend and update. "AI will/won't kill us all" is a flag, not an argument. Session 3.2 shows the fully worked version of this discipline using Carlsmith's explicit credences. Here, start practising it.
A worked mini-decomposition
Try it on a concrete, near-term claim: "A deployed AI agent will cause at least $1 billion in unintended damage within five years." Instead of a yes/no, break it into conditional steps and assign each a credence you can defend and revise:
- Agentic AI is widely deployed with real-world authority (money, code, infrastructure) within five years: say 0.8.
- Given that, some agent pursues an unintended sub-goal at scale (reward hacking or goal misgeneralisation, Sessions 3.1/3.3): say 0.5.
- Given that, at least one instance causes ≥$1B in damage before it is caught: say 0.3.
Multiplying gives about 0.12: roughly one in eight, and here is exactly why. Now you can argue about it link by link rather than as a slogan. Change "five years" to "fifty" and every term shifts. This is the same machinery as Carlsmith's six-premise estimate (Session 3.2), only smaller. The discipline is identical, and it is what the failure-mode essay rewards. (The numbers are illustrative: supply and defend your own, and notice how much the conclusion moves when you do.)
In-class
One sentence, two halves in-class
Write a single sentence: "The strongest reason to take misalignment seriously is … and the strongest reason to doubt it is …" Make both halves strong, with no strawmen on either side. We collect these, surface the themes, and revisit your own sentence in Session 20 to see whether the course changed your mind, and in which direction. Keep a copy.
Questions to bring to class
- Which single link in the "case" chain do you find weakest, and what concrete evidence would raise or lower your confidence in it?
- Is "stochastic parrots" an argument that current models are safe, or just that a particular danger is premature? Could both the parrot view and the misalignment view be partly right?
- Steelman the present-harms critique in one paragraph. Then steelman the reply. Which do you find stronger, and why?
- Hoes & Gilardi find no individual-level crowding-out. Does that address the critics' actual concern, which is often about institutional attention and funding? What evidence would settle that?
- Of Soares' four background claims, which carries your lowest credence, and is the standard objection to it the reason why? If not, state your own objection in one sentence.
Next
Enough abstraction and argument. Sub-session 1.4 grounds all of this in one concrete, recent, peer-reviewed result: a demonstration that safety is unsolved, and unevenly so. It becomes this course's anchor, and it comes with a practical guide to reading the field without being taken in.