What we'll cover
The course has used refusal rates, benchmark scores and attack success rates to study model behaviour. Mechanistic interpretability asks what computation inside the model produces a specified result. Small attention-only models admit detailed circuit analyses; frontier models remain much harder to explain. The worked example on the next page shows how to test one proposed circuit and limit the claim it supports.
Where the taught route goes next
Continue to one circuit and its limits, then the project launch. The source paper develops the mathematics in more depth for independent study.
Three ways to explain a model's behaviour
Suppose a language model correctly continues a repeated sequence. We can describe that result through output tests, attribution or a proposed circuit. Each supplies different evidence.
Levels of explanation
- Black-box / behavioural. Measure the output: how often does the model complete the sequence correctly, and how does that vary with the input? This is what every eval in Session 11 did. It describes the tested behaviour but does not identify the computation that produced it.
- Attribution / saliency. Ask which inputs mattered: gradient-based saliency maps, feature-importance scores, probing classifiers trained on activations. These methods can suggest relevant inputs or representations. A saliency map or probe still needs checks before it can support a causal account.
- Mechanistic. Propose and test a computational account: which components move which information, and what changes when one is intervened on. A successful account explains a specified behaviour in a specified model; it need not describe every computation the model performs.
Mechanistic work can add evidence about what a model computes on the tested inputs. Whether that mechanism generalises beyond them still needs testing. The worked circuit that follows shows how to make and limit such a claim.
Interpretability is the white-box arm of evaluation
Recall the taxonomy from 11.1: evaluations are behavioural (read inputs and outputs only) or internal (read activations and weights). Mechanistic interpretability uses internal access to propose and test a computational account. In Session 11.2, a model can appear incapable on an evaluation while retaining a capability. Internal evidence may help distinguish inability from underperformance, though a probe also needs validation. A model that recognises an evaluation and changes its behaviour may make behavioural evidence unreliable. Internal tests can add evidence, provided the probes and interventions are themselves validated. Current interpretability methods are not a general defence against deceptive behaviour.
The reverse-engineering ambition
A trained network consists of matrices and nonlinearities. With the right access, researchers can record activations and intervene on components. That access makes the system easier to inspect than a biological brain, although it does not make its computations easy to understand. Small attention-only models admit much more complete analyses. As the source paper shows, a one-layer attention-only transformer can be written out almost completely as a set of token-to-token tables read directly from its weights, and a two-layer model's in-context-learning circuit can be reverse-engineered and verified by ablation.
The evidence at frontier scale
For frontier models, researchers often rely on interventions and partial observations rather than a complete account of the computation. Ablations can support causal claims about a tested component in a small model. A feature or attention pattern observed in a larger model may still be correlational until an intervention tests its contribution. Evidence from a small model should not be treated as evidence that the same circuit exists in a much larger model without testing it there. No current method gives a complete account of frontier-model behaviour.
The safety payoffs
Researchers hope internal evidence will help detect deception (Session 3.4), support oversight (Session 8.1) and make independent audits more informative (Session 12). Each use remains uncertain for frontier models. The next page tests a narrow mechanistic claim in a small model and asks what that result licenses us to conclude.
The African-lens angle
An African governance question here is access to model internals: interpretability can operate on a model's weights without needing the developer's training data or cooperation, which could help an institution that has access to the weights of a model it did not train. Whether the features and circuits a model learns are shared across languages, or specific to the high-resource ones, is a real and open question with direct consequences for isiZulu and other low-resource languages. The circuit example does not test how mechanisms vary across languages.
Questions to bring to class
- Give a safety question that a black-box evaluation cannot answer even in principle, and say what a mechanistic explanation would add.
- Why is attribution (saliency, probing) not the same as mechanistic understanding? Construct a case where a saliency map is correct about where and wrong about why.
- Interpretability researchers describe their frontier-scale evidence as more like neuroscience than like decompilation. What does that analogy concede, and which Session-11 failure does it echo?
- Of the three safety payoffs (deception detection, scalable oversight, auditing), which most depends on interpretability working at frontier scale rather than toy scale? Why?
Readings
Before class · required for Session 13
• Olsson et al. (2022), “In-context Learning and Induction Heads” (paper HTML): read the abstract, “Key Concepts” → “Induction Heads”, and the “Summary of Evidence for Sub-Claims” table. Stop there. Be ready to distinguish a behaviour, a candidate head, and a causal intervention.
Used in class
• Olah et al. (2020), “Zoom In: An Introduction to Circuits” (article): we will use its visual vocabulary for features and circuits. No pre-class reading is required.
Optional
• Sharkey et al. (2025), “Open Problems in Mechanistic Interpretability” (paper): for students considering a project in the area.
Next
Continue to one worked induction-head circuit and ask what the evidence does and does not establish. The full source paper develops the QK and OV analysis for independent study.