What we'll cover
Earlier sessions used numbers such as reward-model accuracy and jailbreak success rates. How do we know what those numbers establish? An evaluation can give a precise score while measuring the wrong outcome. We begin with the founding distinction (does the model have a dangerous capability, versus does it have the propensity to use one) and a small taxonomy that organises the rest of the session. An eval is evidence about the tasks and conditions it samples. A result from English prompts does not directly establish performance in isiZulu.
Evaluation as the basis of safety claims
Session 12 considers frontier-safety frameworks that connect evaluation evidence to deployment decisions and safeguards. Some commitments use capability thresholds; the decision rules depend on the policy and its version. For example, Anthropic’s Responsible Scaling Policy (version 3.4, effective 8 July 2026) also includes risk reports and safety roadmaps. A threshold is useful only if its evaluation measures the relevant risk. Shevlane et al. (2023) argue that evaluation is necessary but insufficient for extreme-risk governance: measurements can be incomplete, gamed or invalid. The next three sub-sessions examine these limits.
Capability versus propensity
Shevlane et al. separate two things that everyday talk of "is the model safe?" runs together.
Dangerous-capability versus alignment evaluations
- Dangerous-capability evaluation: does the model have a capability that could cause extreme harm? Can it find a software exploit, design a toxin, autonomously acquire resources? This asks about the ceiling of what the model can do if it tries.
- Alignment (propensity) evaluation: setting aside what it can do, does it have the propensity to apply a capability for harm? Would it, in deployment, actually pursue a misaligned goal or resist correction?
The two come apart in both dangerous directions. A model can be capable but well-behaved (high capability, low harmful propensity), which is the hoped-for case. Or it can be capable and willing, which is the feared case. There is also a subtler case: a model can appear incapable or harmless on the eval while being neither, because it was poorly elicited or because it strategically underperformed. A capability eval measures what the model does under the elicitation methods tested. The best expected score among tested methods is no greater than the best expected score over the feasible methods. A noisy observed score is an estimate, not a guaranteed lower bound or certified ceiling. A propensity eval asks whether the model applies its capabilities for harm under the tested conditions.
The taxonomy
Beyond capability-versus-propensity, three more axes recur, and naming them prevents a common error: reading an eval as proving more than it does.
Three more axes
- Behavioural (black-box) versus internal (white-box). A behavioural eval reads only inputs and outputs; an internal eval reads activations or weights (a deception probe, a logit-lens read). A behavioural result can miss a mechanism that an internal probe detects. This is why access to weights and activations matters when an evaluator claims more than observed behaviour. Behavioural access alone is, in Casper et al.'s phrase, insufficient for many safety claims.
- Static benchmark versus agentic / long-horizon. A static benchmark scores single question-answer pairs; an agentic eval watches a model use tools and act over a long task. Autonomy and self-proliferation require agentic tests to assess the relevant long-horizon behaviour.
- Upper bound versus lower bound. This is the one most often confused. For a defined task distribution and scoring rule, the best expected score among tested methods cannot exceed the best expected score among feasible methods. Measured scores can fluctuate even on a fixed task set because outputs may be stochastic. Choosing the largest noisy estimate introduces selection bias. More elicitation methods can reveal stronger performance, but do not establish a ceiling. Reporting the observed score as a maximum can wrongly clear a model; 11.2 examines this problem.
The 2×2 grid
Cross capability-versus-propensity with behavioural-versus-internal and you get four cells, and "the model passed the eval" means something different in each. Place these and the differences become obvious: WMDP (behavioural capability), an agentic cyber-range exercise (behavioural capability, long-horizon), Apollo’s in-context scheming suite (behavioural capability under assigned goals and incentives), a linear deception probe (internal evidence whose interpretation depends on its training labels and validation), and an isiZulu refusal eval (behavioural propensity, in a non-English distribution). A behavioural result and an internal probe answer different questions. Neither can be upgraded to a broad safety claim without checking the test distribution and the probe itself.
Evaluators and the theory of change
A handful of organisations define the practice, and they map onto the taxonomy. METR and the UK AI Security Institute (renamed from AI Safety Institute in 2025) focus on capability and agentic evals (METR's autonomy task suites; the Institute's open-source Inspect framework). Apollo Research studies deceptive behaviour and scheming capabilities; its instructed scheming tests do not estimate how often a deployed model would scheme. The labs run their own: Anthropic’s Responsible Scaling Policy and Google DeepMind’s Frontier Safety Framework connect evaluation evidence to safeguards and governance decisions under their respective policy versions. The intended path is a chain: an eval produces a measurement, the measurement feeds a risk assessment, the risk assessment triggers a governance action. That chain, eval to threshold to mitigation, is the subject of Session 12, so here we state it and hand it over. This session covers the first link only: making the measurement trustworthy.
The distribution an eval samples
An evaluation is evidence about one thing: the distribution of inputs it actually sampled. A summarisation benchmark tells you about summarisation; a multiple-choice bio test tells you about that question style; an English red-team tells you about English. The moment you generalise "scored safe on the eval" to "is safe," you have made a claim about a wider distribution than you measured, and language coverage must be checked rather than assumed. A claim about safety across languages needs in-language evidence. In 11.3 you will examine the validity of such a claim, and in 11.5 you will audit the Session 9 comparison.
Questions to bring to class
- Describe a model with high dangerous capability but low harmful propensity, then one with low dangerous capability but high harmful propensity. What evidence could support each description, and what would it tell a deployer?
- What does an internal probe add to a behavioural finding that a model did not misbehave when observed? What evidence would you need before trusting the probe?
- Place a dangerous-capability cyber eval, an isiZulu refusal eval, and a deception probe in the capability/propensity × behavioural/internal grid. Which cell is hardest to evaluate well, and why?
- "Necessary but not sufficient." Give one reason evaluation is necessary for safety governance and one concrete way a passed eval can still leave a model unsafe.
Readings
Before class · required for Session 11
• Shevlane et al. (2023), “Model evaluation for extreme risks” (paper HTML): read the abstract, Section 1 (Introduction), the first two paragraphs of Section 4 (Building evaluations for extreme risk), and Section 5.1 (Limitations). Stop there. Bring one example of a capability eval and one reason that a passed eval could still mislead.
Optional
• Casper et al. (2024), “Black-Box Access is Insufficient for Rigorous AI Audits” (paper): for students who want more on behavioural versus internal access.
Next
Sub-session 11.2 builds a real dangerous-capability battery: the four domains, how elicitation can reveal more capability, and how sandbagging (strategic underperformance) can widen the gap between measured and attainable performance.