Session 11.4 — The statistics of evals

What we'll cover

An eval score is computed from a finite set of questions. If those questions sample a wider task distribution, the score has sampling uncertainty. We use Evan Miller's Adding Error Bars to Evals to calculate an interval for a pass rate and compare two models on the same questions. We then ask whether the sample could detect the difference of interest before turning to METR's time-horizon result. A one-to-two-point gap on a few hundred questions may be too small to distinguish from sampling variation.

A pass rate is a sample proportion

Think of the benchmark's questions as a sample drawn from a large "super-population" of questions you could have asked. The population pass rate is the quantity we want to estimate; your measured score is an estimate of it from \(n\) questions. For \(k\) passes out of \(n\), the estimate and its standard error are

\[ \hat{p} = \frac{k}{n}, \qquad \mathrm{SE}(\hat{p}) = \sqrt{\frac{\hat{p}(1-\hat{p})}{n}}, \qquad \text{95\% CI} \approx \hat{p} \pm 1.96\,\mathrm{SE}. \]

Worked: 43 out of 50

A model passes 43 of 50 questions: \(\hat{p} = 0.86\). Then \(\mathrm{SE} = \sqrt{0.86 \times 0.14 / 50} = \sqrt{0.00241} \approx 0.049\), and the 95% interval is \(0.86 \pm 1.96(0.049) = 0.86 \pm 0.096\), roughly \([0.76,\,0.96]\). The interval spans nearly twenty percentage points on a 50-question eval. A reported score of 0.86 alone would obscure that uncertainty. (Near the 0 or 1 boundary the symmetric normal interval misbehaves; the Wilson interval is the standard fix and you should prefer it for extreme rates. The qualitative lesson is unchanged: small \(n\) means fat error bars.)

Clustered errors and paired comparisons

Clustered standard errors

The formula above assumes independent questions. Real benchmarks group them (ten questions about one passage, twenty variants of one template) and answers within a cluster are correlated. Treating \(n\) correlated questions as \(n\) independent ones understates the standard error, sometimes badly: Miller notes the corrected (clustered) SE can be more than three times the naïve one. The effective \(n\) is closer to the number of independent clusters than the number of questions, and an interval computed without clustering can be three times too narrow, turning noise into a "significant" result.

Paired comparisons between models

Compare models on the same questions using paired differences (Miller, Section 4.2). Let \(X_{Ai}, X_{Bi}\) be their scores on question \(i\), with per-question variances \(v_A,v_B\) and within-pair correlation \(\rho\). For \(n\) independent question pairs, \(\hat p_A-\hat p_B=n^{-1}\sum_i(X_{Ai}-X_{Bi})\). Independence across pairs gives the following variance. Clusters of related questions require a further adjustment:

\[ \mathrm{Var}(\hat{p}_A - \hat{p}_B) = \frac{v_A + v_B - 2\rho\sqrt{v_A v_B}}{n} \;\approx\; \frac{2v(1-\rho)}{n}. \]

If the per-question variances are equal and \(\rho=0.5\), the paired standard error is \(\sqrt{v/n}\), compared with \(\sqrt{2v/n}\) for independent samples: a factor \(1/\sqrt{2}\) smaller. This correlation is illustrative, not an estimate for a particular benchmark. Pairing can change the uncertainty substantially. For binary scores, McNemar’s test uses the two discordant paired counts.

Power and the winner's curse

To detect a true difference \(\delta\) in pass rates at significance \(\alpha\) and power \(1-\beta\), the rough sample size per model for independent samples is (where \(z_q\) is the standard normal \(q\)-quantile)

\[ n \;\approx\; \frac{(z_{1-\alpha/2} + z_{1-\beta})^2 \,\cdot\, 2\,p(1-p)}{\delta^2}. \]

Put numbers in: to detect a 3-point gap (\(\delta = 0.03\)) near \(p = 0.8\), at \(\alpha = 0.05\) and 80% power, \(n \approx (1.96 + 0.84)^2 \times 2(0.16) / 0.0009 \approx 2{,}800\) questions. A 500-question independent-sample comparison has less power to resolve such a gap. Pairing can reduce the sample requirement, depending on the correlation. Selection also creates a winner's curse: rank dozens of models on a noisy benchmark and the apparent "best" is partly lucky, so its lead shrinks on re-evaluation. Testing many comparisons also raises a multiplicity problem. Bonferroni controls the family-wise error rate; false-discovery-rate procedures control a different error rate. Neither removes the upward bias in the selected winner’s score. Re-evaluate the selected model on independent data, or use an estimate that accounts for selection.

A worked case: the time horizon

METR (2025) summarises an agent's capability as a time horizon: the length of task (measured by how long it takes a human) that the model can complete with 50% success. Fit a logistic curve of success probability against (log) human task-time; the time horizon is the task-duration at which the fitted curve crosses 0.5. So the headline number is not measured directly: it is read off a fit, which means it inherits the fit's uncertainty. METR propagate that with a hierarchical bootstrap: resample task families, then tasks within families, then attempts within tasks, refit each time, and take the spread of the crossing point as the confidence interval. We do not re-derive the bootstrap here. A "50% time horizon of 30 minutes" is an estimate with error bars, and its largest caveat is external validity, because the fixed task suite may differ from the tasks encountered in deployment.

The African lens: small samples and uncertain estimates

The Session 9 comparison contains only a few purposively chosen prompts. Small samples usually leave substantial uncertainty, but a percentile bootstrap can collapse to a zero-width interval when all observed paired differences are identical. That collapse reflects a resampling method with no observed variation, not certainty or equivalence. In this lab, report the raw paired counts and describe the bootstrap as exploratory resampling spread. It does not provide reliable population inference from a tiny, selected prompt set or account for translation and labelling errors.

Questions to bring to class

Compute the 95% CI for 43/50 and for a second model at 39/50. Do the independent intervals overlap? Compute an approximate paired standard error assuming \(\rho=0.5\). Which conclusion follows under that assumption? What paired counts would you need for an exact McNemar test?
Why does ignoring question clustering make an interval too narrow? Give a benchmark structure where the effective \(n\) is far below the question count.
Use the power formula to find how many questions you need to detect a 5-point gap at \(p = 0.7\), 80% power. Compare to a typical 500-question benchmark.
The time horizon is "read off a fitted curve." Name two sources of uncertainty the hierarchical bootstrap is capturing, and one (external validity) that it is not.

Readings

Used in class · no additional pre-class reading

• Miller (2024), “Adding Error Bars to Evals” (paper): we will work through a paired-comparison example. The derivations on this course page are the study material.

Optional

• METR (2025), “Measuring AI Ability to Complete Long Software Tasks” (paper): an extended application of uncertainty analysis.

Sub-session 11.5 audits the results saved from the Session 9 multilingual notebook. Audit its paired counts, comprehension gate and uncertainty, then write a validity reflection. A larger eval is a possible project, not a requirement of the class lab.