What we'll cover
"AI safety" gets used to mean everything from chatbot profanity filters to human extinction. Used that broadly, the term is nearly useless. This sub-session sharpens it. We give a working definition, separate the three kinds of risk the field worries about, and introduce the distinction, safety versus capabilities, that organises almost everything that follows. We then map the technical field (the map is also the structure of this course), sketch how the field came to exist, place this course among the others you might encounter, and say clearly what technical AI safety is not.
Mandatory readings
• Amodei, Olah, Steinhardt, Christiano, Schulman & Mané (2016), "Concrete Problems in AI Safety" (arXiv:1606.06565): short, foundational, and still the most useful map of the empirical field; read §1–2 and the openings of §4, §5 and §7 (reward hacking, scalable oversight, distributional robustness). ≈3,000 words.
• Hendrycks, Mazeika & Woodside (2023), "An Overview of Catastrophic AI Risks" (arXiv:2306.12001): the four-category risk taxonomy; read §1 and the openings of §2–5 (Malicious Use, AI Race, Organizational Risks, Rogue AIs). ≈3,500 words.
Total mandatory load: ≈6,500 words.
Optional readings
• Ngo, Chan & Mindermann (2022; ICLR 2024), "The Alignment Problem from a Deep Learning Perspective" (arXiv:2209.00626): the modern, LLM-era framing we mostly use.
• Russell, S. (2019), Human Compatible: Artificial Intelligence and the Problem of Control (Viking; ISBN 978-0-525-55861-3): the value-alignment reframing, very readable.
• CAIS (2023), "Statement on AI Risk" (safe.ai/work/statement-on-ai-risk): one sentence; read who signed it.
A working definition
Technical AI safety is the study of how to build AI systems we can rely on to do what we intend, and not what we don't, and how to know whether we have succeeded.
Three verbs in that sentence carry the weight. Build, because this is an engineering and research discipline about real systems, not only a branch of ethics or policy. Intend, because the hard part is that "what we intend" is rarely something we can write down exactly (Session 3.1). And know, because even a system that behaves well may not be safe if we cannot verify why it behaves well. That theme drives the interpretability and evaluation weeks.
Technical, sociotechnical and governance
This course is "technical" AI safety: the algorithms, training methods, evaluations and model internals. That is not a claim that the technical layer is the only one that matters. AI risk is profoundly sociotechnical, and we treat governance (Session 12), the physical and economic substrate (Session 4), and ethics (Session 8) as first-class topics. The "technical" label marks where our attention is concentrated, and where this cohort's advantage lies.
Three kinds of risk
Most concrete worries fall into one of three buckets. The buckets call for different technical responses, so muddling them wastes effort.
Misuse
The system works exactly as intended, but a person directs it at harm: cyber-offence, bioweapon "uplift", fraud, targeted disinformation, surveillance. The model is a capable tool; the danger is the human wielding it. The technical responses are refusal training, dangerous-capability evaluations, structured access and monitoring.
Misalignment
The system pursues something other than what we intended. Either we specified the wrong objective (outer misalignment) or it learned an unintended one (inner misalignment). No malicious user is required. This is the distinctive, hardest, most new problem, and the centre of this course (all of Week 2).
Systemic / structural
Harms that emerge from how AI is deployed across society: concentration of power, labour displacement, mass surveillance, environmental cost, erosion of the shared information ecosystem. No single model fails; the harm is a property of the system and its incentives.
Hendrycks' four categories
Hendrycks, Mazeika & Woodside's "An Overview of Catastrophic AI Risks" (2023) uses four categories, which map cleanly onto the three above. Malicious use is roughly misuse. Rogue AIs is roughly misalignment. The AI race and organizational risks are two flavours of systemic risk: competitive pressure that erodes caution, and ordinary institutional failure inside the labs building these systems. Either taxonomy is a thinking tool. Its value is that it forces the question: which kind of danger is this, and what would actually reduce it?
Overlapping categories
A misaligned model, in the hands of a malicious user, inside an organisation racing a competitor and cutting safety corners, is all three at once. That is also the realistic case. The three-way cut exists for thinking, not filing. When someone says "AI is dangerous", first ask which danger they mean. The mitigations differ and often pull in different directions.
The organising idea: safety versus capabilities
The widening gap
Capabilities are what a system can do: solve problems, write code, pass exams, act in the world. Safety is whether we can rely on it to do what we want, and whether we would know if it did not. The central empirical worry of the field is a widening gap. Capabilities are improving fast and predictably (Session 2's scaling laws). Safety, meaning our ability to specify objectives, verify behaviour and retain control, is not keeping pace.
Almost every topic in this course tries to close that gap from the safety side: better specification (alignment methods, Weeks 3 and 4), better verification (evaluations and interpretability, Weeks 6 to 8), better control (robustness and the control agenda, Week 5). Keep the distinction in hand; we test new ideas against it all term.
Differential progress
Not all research is equal from a safety standpoint. Work that advances safety faster than raw capability narrows the gap. Work that does the reverse widens it, even if well-intentioned. This idea is called differential technological development. It is why some safety researchers are careful about which capabilities their work might push forward, and it recurs when we discuss how entangled interpretability and evaluations are with capabilities.
A map of the technical field
The technical problems cluster into a handful of families, and the course is structured around them. Two reference framings, a decade apart, help orient you.
The classic list: "Concrete Problems in AI Safety" (Amodei et al., 2016)
The paper that made the field legible to mainstream ML named five concrete problems, all framed around an agent and its reward: (1) avoiding negative side effects, (2) avoiding reward hacking, (3) scalable oversight, (4) safe exploration, and (5) robustness to distributional shift. Nearly a decade on, problems (2), (3) and (5) are central pillars of this course (Sessions 3.1, 8, 9). The framing has held up remarkably well.
The modern framing
Ngo, Chan & Mindermann's "The Alignment Problem from a Deep Learning Perspective" (2022; ICLR 2024) recasts these problems for the era of large pretrained models: situationally-aware policies, learned objectives, deception. That is the version we actually study. Across both framings, the technical work clusters into four families, and the course is built around them:
- Alignment: specifying and instilling the right objective (RLHF, RLAIF, Constitutional AI, scalable oversight). Weeks 3 and 4.
- Robustness and control: holding up under adversarial pressure, and staying controllable even if the model is not cooperative. Week 5.
- Evaluation and governance: measuring what a system can and will do, and the mechanisms that make a safety policy enforceable. Week 6.
- Interpretability: reading a model's internals to understand why it does what it does, since behaviour alone is insufficient. Weeks 7 and 8.
The field's three phases
Concern about machines pursuing the wrong goals is old. Wiener and the cybernetics tradition worried about it, and Asimov wrote fictional "laws" about it. But technical AI safety as a research field is recent, and it has three rough phases.
First, a philosophical and theoretical phase. Bostrom's Superintelligence (2014) and Stuart Russell's reframing of AI's goal in Human Compatible (2019) set out the conceptual problem. Second, an empirical-ML phase, kicked off by "Concrete Problems" (2016), which translated it into experiments on real systems. Third, the current large-model phase, driven by the surprising capabilities of LLMs since about 2020. It made once-speculative concerns measurable, deception and situational awareness among them, and brought governments in: the UK and US AI Safety Institutes (both renamed in 2025: the UK's is now the AI Security Institute, the US's the Center for AI Standards and Innovation, CAISI), frontier-lab safety policies, and the May 2023 CAIS one-sentence Statement on AI Risk. Signed by many of the field's founders, it placed extinction-level risk "alongside other societal-scale risks such as pandemics and nuclear war". You are studying the field in its most empirical, and most contested, moment.
Where this course sits
Relative to the courses you may have heard of
There are several excellent open courses, and we draw on all of them. BlueDot's AI Safety Fundamentals is a conceptual, reading-group curriculum, good for breadth and framing. ARENA (the Alignment Research Engineer Accelerator) is a hands-on engineering bootcamp, and the source of several of our labs. Hendrycks' "Intro to ML Safety" is the closest technical analogue and supplies much of our backbone and problem sets. Grosse's Toronto course gives us the idealised-to-empirical sequencing and the project structure.
This course is its own synthesis. It is pitched at mathematically minded honours students, so it is more formal where formality helps (see the optional derivation appendices). It is built around a research project. And it is grounded in an explicitly African framing the Northern courses do not carry. We point you to the others as you go; none is a prerequisite.
What technical AI safety is not
Five things to keep straight
- Not only ethics. Ethics matters (Session 8), but "is it good?" is a different question from "does it do what we intended, and can we tell?".
- Not only policy. Governance matters and is itself technical (Session 12), but this course's core is the systems, not the statutes.
- Not science fiction. We take seriously arguments tied to evidence about real systems. A claim that cannot be tested empirically or formally gets the same scepticism as any other untestable claim.
- Not solved. No technique in this course "solves alignment". Each manages some part of the gap, and being clear about what remains open is the aim.
- Not anti-AI. Studying failure modes is how engineering disciplines make powerful technologies usable. Aerospace safety is not anti-flight. The aim is AI that is beneficial and reliable, including for the people the technology currently serves least.
Questions to bring to class
- Take a recent AI-related news story. Which of the three risk types is it: misuse, misalignment, or systemic? Is it cleanly one, or several at once?
- Give an example where improving a system's capabilities made it less safe, and one where a safety intervention also improved capability. What does each imply for "differential progress"?
- Of Amodei et al.'s five concrete problems, which do you expect to be hardest, and why? Which feels most relevant to systems you've actually used?
- The CAIS statement compares AI risk to pandemics and nuclear war. Is that comparison illuminating or misleading? (We return to this in 1.3 and Session 20.)
Next
We've named the problem and mapped the field. Sub-session 1.3 asks whether it's actually worth worrying about: the strongest version of the case for taking misalignment seriously, and the strongest arguments against, including a critique with a real Global South dimension.