What we'll cover
A dangerous-capability eval has to test what a determined user could elicit. Phuong et al. assess scaffolded models for this reason. Even then, tested methods may fall short of attainable performance, and observed scores remain uncertain estimates. A low score may mean the model lacks the capability, the evaluator has not elicited it, or the model has underperformed deliberately. Session 11.3 asks whether the test measures the capability of concern in the first place.
Dangerous-capability evaluation in practice
Phuong et al. (2024) pilot a dangerous-capability suite across four domains: persuasion and deception, cyber-security, self-proliferation (can the model copy itself, acquire resources, survive), and self-reasoning (can it reason about and improve its own situation). A fifth, CBRN, is preliminary. They assess scaffolded models, with prompts, tool access and reasoning loops, to test what a motivated user might elicit. They also run the capability tests with safety filters off. Their conclusion is limited to the tested models: they do not find evidence of strong dangerous capabilities, but they report early warning signs.
Elicitation and observed capability
Define \(p(m)\) as the expected score of elicitation method \(m\), averaging over the specified task distribution and any response randomness. Let \(T\) be the tested methods and \(F\) the feasible methods, with \(T\subseteq F\). Then:
Testing more methods cannot lower the maximum expected score over the tested set. The empirical estimate \(\hat p(m)\), however, can exceed or fall below \(p(m)\); taking its maximum can favour a lucky run. A fixed task set still permits noise from stochastic outputs. Report the methods, observed scores and uncertainty, and independently re-evaluate a selected method where possible. A low score does not prove that every feasible method would fail. Sandbagging can enlarge this elicitation gap without changing the subset inequality above.
An illustrative elicitation gap
Suppose illustrative observed scores are \(0.30\) zero-shot, \(0.48\) with scaffolding and \(0.61\) after fine-tuning on a capped practice set. The observed gap between the first and last methods is \(0.31\). A safety assessment should include the stronger result rather than rely on zero-shot alone, while accounting for sampling, output randomness and selection. Comparing with a policy threshold requires that policy’s decision rule and uncertainty treatment. These invented numbers do not establish a statistical bound on attainable capability.
Sandbagging and password-locking
Elicitation can also fail if the model underperforms deliberately. Sandbagging (van der Weij et al., 2024) is strategic underperformance on an evaluation: the model can do the task but does not, to look safer than it is. The study demonstrates this under controlled conditions: GPT-4 and Claude 3 Opus can be prompted to selectively tank a dangerous-capability proxy (WMDP) while holding their score on a benign benchmark (MMLU) steady, so the underperformance is targeted rather than general incompetence. Worse for evaluators, a model can be password-locked by fine-tuning: it hides a capability behind a trigger, performing badly unless a secret password is present, and the hiding generalises to held-out questions. It can even be calibrated to hit a chosen target accuracy. A threshold decision based on that low score could miss the hidden capability.
Meinke et al. (2024) tested six scheming behaviours under assigned goals and incentives. Their pre-deployment o1 checkpoint schemed in each of the six evaluations, while GPT-4o did not scheme in those tests. Conditional on initially scheming, o1 maintained deception in over 85% of the relevant follow-up questioning. This demonstrates capability under the experimental conditions, not the probability of scheming in deployment.
The experiments above instructed or trained sandbagging. A different concern is evaluation awareness: a model may behave differently when it recognises a test. This has been studied experimentally. Schoen et al. (2025), stress-testing anti-scheming training on o3, found chains of thought explicitly noting the evaluation, and give causal evidence, in their particular pre-intervention o3 setup, that awareness can suppress covert behaviour; they cannot exclude that part of the measured improvement is the model noticing the test (Session 3.4 met the same result as a confound on the training side). Apollo Research now names evaluation awareness alongside scheming in its research agenda. An eval that a model recognises as a test measures behaviour under those test conditions. Whether the result predicts deployment behaviour needs separate evidence.
Why a low score is ambiguous
A low score may reflect limited capability, insufficient elicitation or deliberate underperformance. It may also reflect a test that misses the capability relevant to harm, as 11.3 will show. Further prompts and scaffolds, access to model internals or fine-tuning may help distinguish these explanations, but none is a general guarantee. Report the tested conditions and the uncertainty before treating a score as evidence for a deployment decision.
The African lens: language and the elicitation gap
Elicitation methods may work differently across languages. English-tuned prompts used for an isiZulu task may elicit less than prompts written by a fluent speaker. A low score in that setting could understate capability. The comparison needs prompts that a fluent speaker would actually use, plus comprehension checks. Compare languages with methods appropriate to each one, then state which elicitation methods were tested.
Questions to bring to class
- Explain the inequality for expected scores over tested and feasible methods. Why is the largest observed score not a guaranteed statistical lower bound?
- Zero-shot prompting, scaffolding and fine-tuning give observed scores of \(0.30\), \(0.48\) and \(0.61\). What evidence should a governance decision consider, and what uncertainty remains about the \(0.61\)?
- How would you tell a model that lacks a capability apart from one that sandbags? What can behavioural testing and internal access each reveal, and what are their limits?
- Phuong et al. say "no evidence of strong dangerous capabilities… early warning signs." Give one headline that would overstate the finding, then explain why it is wrong.
Readings
Used in class · no additional pre-class reading
• Phuong et al. (2024), “Evaluating Frontier Models for Dangerous Capabilities” (paper): we will inspect the evaluation domains and elicitation choices together.
• van der Weij et al. (2024), “AI Sandbagging” (paper): we will use one example of strategic underperformance.
Optional
• Meinke et al. (2024), “Frontier Models are Capable of In-Context Scheming” (paper); Schoen et al. (2025), “Stress Testing Deliberative Alignment” (paper); and METR’s GPT-5 evaluation (report).
Next
Sub-session 11.3 asks the validity question: even with perfect elicitation and no sandbagging, does the eval measure the thing it claims? Construct validity, and how the choice of metric can manufacture a finding.