Week 1 — orientation and foundations
Welcome. This first week sets things up before the safety content starts. It covers how the course runs (1.1), what technical AI safety is (1.2), the case for the field and the strongest arguments against it (1.3), and a concrete demonstration that the problem is unsolved, especially in an African context (1.4). Session 2 then rebuilds the deep-learning foundations the rest of the course depends on.
This session's sub-sessions: 1.1 how this course works · 1.2 what technical AI safety is · 1.3 the case and its critics · 1.4 safety isn't solved.
What we'll cover
This course is for anyone interested in AI safety, though it is aimed at a more technical audience. Before any content, we want to be clear about the deal between us: what the course assumes you already know, what you should be able to do by the end, how your work is graded, and how to study a fast-moving, contested field without drowning in it or writing it off. This sub-session is that deal. None of it is examinable, but reading it carefully will make the next 12 weeks much easier.
Who this course is for
You come from a range of backgrounds. The course is offered in the Department of Mathematics and Applied Mathematics and can be taken by students from other science departments, particularly computer science.
The prerequisites are modest. You understand what a neural network is (a parametrised function fitted to data by minimising a loss). You are comfortable with linear algebra, probability and calculus at second or third-year level. You can read and modify a Python script. That is the whole list.
What you don't need
No prior exposure to AI safety, alignment or the research literature. No machine-learning research experience. No GPU of your own: every graded lab runs on free Google Colab. If you have never read an arXiv paper, Session 1.4 teaches you how.
Where the cohort varies
Some of you know deep learning well and some of you do not. Session 2 therefore rebuilds the transformer from neurons up, at a mathematical level, rather than assuming it. If you already know transformers cold, treat Session 2 as consolidation and shared vocabulary. If you are rusty, it is your safety net.
What you'll be able to do
By the end you should be able to state the core alignment problems precisely, read and critique a safety paper, run interpretability and evaluation experiments in code, reason about governance mechanisms, carry out a small piece of safety research, and place all of it in an African context.
The shape of the course
The course runs 12 weeks with 2 sessions a week, so 24 sessions in total. They fall into 5 parts. We start with idealised, formal questions, move to messy empirical practice, and end with your own research.
Part 1 · Foundations and the problem
What technical AI safety is. The deep-learning and scaling foundations. The core alignment problem: specification, instrumental convergence, inner misalignment, deception. The physical substrate of compute, energy and minerals that all of it rests on.
Part 2 · Training and aligning
How a frontier model is actually built (pretraining, supervised fine-tuning, RLHF), then the methods meant to align it: RLAIF, Constitutional AI, scalable oversight, and the question of whose values.
Part 3 · Robust, controllable, evaluable
Adversarial robustness and jailbreaks. Unlearning and the AI-control agenda. Evaluations and technical governance: how we measure what a system can and will do, and what makes a safety policy enforceable.
Part 4 · Mechanistic interpretability
Opening the model up (features, circuits, superposition) with hands-on TransformerLens labs. Behaviour alone cannot tell us whether a model is safe, a point Week 2 foreshadows.
Part 5 · Research project
A scoped, Colab-feasible safety project. You produce a short paper plus code, present it to the class, and review each other's work. Roughly the final third of the course.
The two-act structure
The first half gives you formal objects you can reason about cleanly: objectives, optimisers, proxies, agents, theorems about power-seeking. The second half turns to the real systems we actually have, where those clean ideas meet noisy practice, partial evidence and engineering trade-offs. This ordering plays to your strengths. You start where you are most at home, with definitions and proofs, and move toward where the field actually lives, in experiments and judgement under uncertainty. The structure is borrowed, with thanks, from Roger Grosse's alignment course at Toronto.
Four through-lines
Four ideas recur in almost every session. Watch for them. The course is, in a sense, 24 variations on these.
- The safety–capabilities distinction. What a system can do (capabilities) is not the same as whether we can rely on it to do what we intend (safety). The organising worry of the whole field is the widening gap between the two. We test almost every idea against it.
- From idealised to empirical. Formal models of agency first, then the messy systems we actually have. Each half informs the other. The theory tells you what to look for; the experiments tell you whether it is there.
- African technical AI safety. This is a thread, not a single week. Frontier models are measurably less safe in African languages (Session 1.4). Compute, energy and data sovereignty shape who can build and govern these systems (Sessions 4 and 12). Relational ethics reframes the alignment target (Session 8). "Whose safety, whose risks, whose values?" runs throughout.
- Intellectual honesty. The field disagrees with itself, sometimes sharply, and a good deal is uncertain. We steelman the sceptics (Session 20), hold beliefs at the confidence the evidence supports, and treat "I don't know, and here's why" as a respectable answer. Often it is the correct one.
Assessment
The assessment rewards understanding, calibrated judgement, and the ability to do safety work rather than exam recall. Weights below are indicative and may be adjusted to faculty norms.
Components
| Component | Weight | What it is |
|---|---|---|
| Weekly labs / problem sets | 25% | The coding labs (TransformerLens, UK AISI Inspect, robustness, evals) and short problem sets. Graded on completion and correctness. Resubmission is allowed, because we are after mastery, not one-shot performance. |
| Paper presentation + critiques | 15% | You present one paper to the class. Go beyond summary: what does it claim, what is the evidence, does it hold up? You also submit short written critiques of peers' chosen papers before the relevant seminar. |
| Failure-mode analysis | 10% | A mid-course essay of about 1,500 words analysing one alignment failure mode rigorously (announced in Session 3.4). An African or Global-South failure mode is one sanctioned track. |
| Research project | 40% | Proposal, then a paper of 6 to 8 pages plus code, in small teams. Marked by an itemised, per-section rubric (abstract / method / experiments / limitations / creativity) plus a positioning-and-context criterion. |
| Peer review of projects | 10% | Anonymised reviews of 2 peers' final projects. This teaches you to review, and adds feedback bandwidth. |
Grading the African lens
Every project carries a one-paragraph context statement: who does this help, whose risk does it address, and how, if at all, does it apply in an African context? "Not applicable, because…" is a perfectly good, gradable answer for a pure-theory project. What matters is that you have thought about it rather than performed a gesture. The failure-mode essay and several project tracks offer strong African-context options (Sessions 9, 11 and 18). None is compulsory.
The course's AI-use policy
You may use AI tools in your work, and you must disclose how. This is not box-ticking. In a course about the limitations and failure modes of AI, using AI uncritically is a contradiction. The skills this course trains are exactly the skills that make AI assistance safe to use: verifying claims, reading code you did not write, checking citations, noticing sycophancy and reward-hacked outputs. These very pages were drafted with AI and then reference-checked. That is the standard we hold you to. Submitting unverified AI output as your own analysis is an integrity violation, and it demonstrates that you have missed the point of the course.
The weekly rhythm
Two sessions a week
Usually one concept or seminar session and one lab or working session. Read the assigned material before class. Sessions assume it, and the seminars only work if you arrive ready to argue.
Labs run on Colab
Every graded lab runs on free or Pro Google Colab, with no special hardware needed. A few GPU-hungry labs, such as full RLHF or adversarial-training runs, are marked optional. Nothing graded depends on hardware you may not have.
This site is home base
All sub-sessions, readings and labs live here. The "← Previous / ⌂ / Next →" bar at the foot of each page walks you through in order; the ⌂ button returns to the contents page. Optional deep-dive appendices, like the derivation linked from Session 3.1, sit off to the side.
Study habits
A few habits, learned from watching how this material tends to go wrong for students.
- Read critically. Section 1.4 gives you a checklist: claim versus evidence, what is the eval, who is the population, what would falsify it. Apply it to every reading, including these pages.
- Do the labs. The difference between "I understand attention" and "I found an induction head in GPT-2" is the difference between this course working for you and not. The labs are where the abstractions become real.
- Take a side in the seminars. Several sessions ask you to argue a position you may not hold. That is the fastest way to discover which premises your position rests on.
- Keep a running map. Almost every later technique answers a weakness exposed earlier; the alignment problem in Week 2 motivates everything after it. Note the connections, because the assessment rewards you for seeing them.
- Hold uncertainty. Resist the pull to either doom or dismissal. On many questions here the position is a probability, not a verdict, and you will be assessed on the quality of your reasoning.
What makes this African technical AI safety
Safety for whom
The technical core of AI safety (alignment, interpretability, control) was largely developed in a handful of Northern labs, and most courses inherit that vantage point. This one does too, but it adds a question those courses rarely ask: safe for whom? The anchor, in Session 1.4, is a concrete, peer-reviewed result. Frontier models' safety guardrails fail far more often in isiZulu and other low-resource languages than in English. That is not an ethics footnote. It is a robustness and evaluation problem at the technical core of the field, and it lands on this continent first. We return to it as a lab (Session 9), an evaluation project (Session 11), and a full session on AI safety from the Global South (Session 18), alongside compute and data sovereignty (Sessions 4 and 12) and relational ethics (Session 8). You will not be asked to pretend an African angle exists where it does not. You will be asked, repeatedly, to check.
Practicalities and a standing caveat
- Set up Colab now. A Google account is the only requirement; the first lab (Session 2.5) walks you through
pip installand loading a model. Save your notebooks: you submit them. - Where things live. Readings are linked from each page. Primary sources are preferred over summaries. Where a reading is paywalled, an open preprint is given.
- Errors are possible. These materials are AI-assisted and reference-checked, but mistakes in links, citations or statistics can survive. Treat every number as something to verify, and if you spot an error, email jonathan.shock@uct.ac.za. Finding one is good practice.
To do before the next session
- Confirm you can open a Google Colab notebook and run a trivial cell.
- Skim the contents page so you have a mental map of the 12 weeks.
- Come to Session 1.2 with a one-line answer to: "What do you currently think 'AI safety' means?" We'll sharpen it together.
Next
Now that you know how the course runs, sub-session 1.2 asks the obvious question: what is technical AI safety? What are the kinds of risk, what is the one distinction that organises the field, and what is it not?