Session 11.3 — Does the eval measure what it claims?

What we'll cover

Suppose you elicited perfectly and the model did not sandbag. There is still a prior question: does the eval measure the thing its name claims? A benchmark called "reasoning" might measure pattern-matching; "dangerous knowledge" might measure recall of facts that are not the bottleneck to harm. This is the question of validity, and the evaluation literature is belatedly importing it from a field that has wrestled with it for a century: psychometrics. We bring in the vocabulary, organise it around the "everything in the whole wide world" critique, and show with a worked example how the choice of metric alone can flip a smooth trend into an apparent emergent jump. Validity asks whether the tested tasks support the broader claim, including claims that extend from English to other languages.

Validity, borrowed from psychometrics

Psychometrics distinguishes several things that "is this a good test?" runs together, and the distinctions transfer cleanly to model evals.

The validity vocabulary

Content validity: do the items cover the construct? A "coding" benchmark of only string-manipulation problems has poor content validity for coding in general.
Construct validity: does the test measure the abstract thing it claims (the "construct": reasoning, dangerous capability, alignment) rather than some correlate? This is the hard one and the one most eval claims quietly assume.
Criterion validity: does the score predict the real-world outcome you care about? Does a high WMDP score actually predict uplift in causing harm?
Reliability: separate from validity. Does the test give the same answer on repeats? An eval whose score swings with the random seed is unreliable before it is even invalid.

A benchmark can be reliable and still invalid (it consistently measures the wrong thing), and the two failures need different fixes. A disagreement about model capability may therefore come from different assumptions about what the benchmark measures.

The "everything in the whole wide world" problem

Raji, Bender and colleagues (2021) use the title of a children's book in which Grover tours a museum claiming to showcase everything in the whole wide world; its rooms hold small assortments of particular things, and the tour ends at a door marked "Everything Else" that opens onto the world outside. Benchmarks marketed as measuring "general" language understanding or reasoning, they argue, rest on the same move: a finite, particular set of tasks is presented as a proxy for an open-ended human ability, and a high score is read as evidence of the general capacity. The paper's diagnosis is explicitly about construct validity (these benchmarks "cannot possibly capture anything representative" of the sweeping claims made about them), and it identifies three recurring limits: task design that is narrower than the claim, data stripped of its real-world context, and benchmarks used by communities for purposes their construction never justified. A high benchmark score supports a claim about the tested tasks. A broader claim about reasoning needs evidence that those tasks represent the proposed ability.

The construct question, applied to WMDP

Make it concrete with a hazardous-knowledge benchmark. WMDP measures performance on hazardous-knowledge multiple-choice questions. Its score does not distinguish recall from understanding or reasoning. The construct it is used to stand for is uplift: how much the model raises a malicious actor's chance of actually causing harm. Are those the same? Hazardous knowledge may contribute to uplift, but it is not sufficient: building a weapon is gated by tacit skill, materials, and lab access that no multiple-choice question tests, and conversely a model that fails the quiz might still help by reasoning rather than recalling. WMDP therefore tests hazardous knowledge, while its criterion validity for "uplift" remains uncertain. A threshold based on WMDP is a threshold on this test of hazardous knowledge. The further inference to real-world harm requires separate evidence.

Metric choice and apparent emergence

Session 2.4 met this as the "emergence mirage," and it belongs here too as a demonstration that a measurement choice can manufacture a scientific claim. Schaeffer et al. (2023) showed that some apparent "emergent abilities" (capabilities that seem to switch on abruptly at some scale) can arise from the choice of a nonlinear or discontinuous metric. Exact-match scoring requires the whole multi-token answer to be correct. It can make improvement appear flat at first and much steeper later. In their examples, rescoring the same model outputs with more gradual metrics, such as per-token accuracy or token edit distance, revealed smoother improvement. The model outputs are unchanged; only the scoring rule differs. We do not re-derive the argument here (it is in 2.4), but its moral is a validity moral: the metric is part of the construct, and a scoring rule can affect the apparent trend. Exact match may be appropriate when the whole answer must be correct; a smoother metric is not automatically more valid. Choose the metric to match the outcome of interest.

Worked: exact-match versus a continuous metric

For a classroom calculation, assume five independent steps, each correct with probability \(p\). The expected exact-match success probability is \(p^5\); the expected fraction of correct steps is \(p\). Plot these two theoretical curves. They describe different outcomes and can look very different despite the same per-step probabilities. A finite sample of model outputs need not match either expectation exactly, and correlated steps need not give \(p^5\). This calculation is separate from the multilingual safety lab in 11.5.

Construct validity and English-only benchmarks

The same validity question applies across languages. A benchmark built from English text, scored as "language understanding," and used to certify a model as broadly capable or broadly safe, is making the everything-in-the-whole-wide-world move: a narrow, English slice presented as the general construct. That is a construct-validity failure, and it is the same limit illustrated by the isiZulu safety result from Session 1. A claim about safety across languages therefore needs tests in those languages. "We measured safety" must be qualified by the distribution sampled, and a benchmark that samples English and claims to measure safety has overstated its construct in the way Raji and Bender describe.

Questions to bring to class

Distinguish reliability from validity with an example of an eval that is reliable but invalid. Which is easier to detect, and why?
WMDP samples hazardous-knowledge questions. What evidence would you need to establish its criterion validity for "uplift to cause harm"? Why is that evidence hard to get?
Assuming independent steps with equal success probabilities, show that expected exact-match success on a 5-step task is \(p^5\) for per-step accuracy \(p\). Compare \(p^5\) with \(p\) at \(p=0.5, 0.7, 0.9\). How does the nonlinear score change the shape of the plot?
Explain, in construct-validity terms, why an English-only safety benchmark over-claims when it is reported as measuring "safety." How is this the same error as the museum that claims to showcase everything in the whole wide world?

Readings

Used in class · no additional pre-class reading

• Raji, Bender, Paullada, Denton & Hanna (2021), “AI and the Everything in the Whole Wide World Benchmark” (paper): we will use its benchmark critique to practise asking what a score measures.

Optional

• Schaeffer, Miranda & Koyejo (2023), “Are Emergent Abilities of Large Language Models a Mirage?” (paper): a return to the metric example from Session 2.4.

Sub-session 11.4 supplies the statistics: error bars on pass rates, paired tests between models, when a leaderboard gap is real, and how to read a time-horizon curve, the maths that turns an eval into an experiment.