What we'll cover
Scaling laws (2.3) describe a smooth fall in loss, yet headlines describe abilities that appear "all at once". This sub-session separates what we can measure (how fast compute and capability have grown) from what is contested: whether new abilities emerge discontinuously at scale, or only seem to because of how we score them. We work the metric mechanism behind that dispute with real numbers, place it inside the deeper tension between predictable loss and surprising behaviour, and draw the safety conclusion: where forecasting fails, you have to measure directly.
Mandatory readings
• Wei, J., et al. (2022), "Emergent Abilities of Large Language Models" (TMLR; arXiv:2206.07682): read as a pair with the Schaeffer rebuttal below; read §1–3 and the eight emergence curves in Figure 2. ≈1,500 words.
• Schaeffer, R., Miranda, B. & Koyejo, S. (2023), "Are Emergent Abilities of Large Language Models a Mirage?" (NeurIPS 2023; arXiv:2304.15004): read §1–2 (the alternative explanation) and the meta-analysis in §4. ≈1,500 words.
• Kwa, T., West, B., et al. (2025), "Measuring AI Ability to Complete Long Software Tasks" (METR; NeurIPS 2025; arXiv:2503.14499; live data at metr.org/time-horizons): skim only, enough to understand the key claims: what the 50%-time-horizon metric is, and roughly how fast it has grown. Be sceptical of how meaningful "task length" is as a measure: the tasks are software tasks with clean success criteria, and METR's own limitations note is candid about how far it generalises. The measurement has already been revised once (Time Horizon 1.1, January 2026, on a larger task suite) and continues to be updated, so treat any specific doubling time as provisional. ≈500 words at skim depth.
• Favaro, M. & Clark, J. (2026), "When AI Builds Itself" (Anthropic): a frontier lab's measured evidence that the self-improvement loop is part-closed: over 80% of Anthropic's production code now written by Claude, task horizons doubling every ~4 months (against METR's earlier ~7), and a proposal to preserve the option of a coordinated pause. Read the opening and the internal-evidence sections; skim the three-futures scenarios. Focus on understanding the key figures: for each one, be able to say what is being measured, over what period, and what would have to be true for the trend to continue. Read it with the Session 1.4 checklist in hand: this is evidence from the actor with the most at stake, which cuts both ways. ≈2,500 words at this depth.
Total mandatory load: ≈6,000 words (two close reads, two skims).
Optional readings
• Ganguli, D., et al. (2022), "Predictability and Surprise in Large Generative Models" (arXiv:2202.07785): the predictable-loss / surprising-capability tension.
• Srivastava, A., et al. (2022), "Beyond the Imitation Game" (BIG-Bench; arXiv:2206.04615): the benchmark suite behind the emergence claims.
• Sevilla, J., et al. (2022), "Compute Trends Across Three Eras of Machine Learning" (arXiv:2202.05924): and Epoch AI (epoch.ai) for live trends.
• Davidson, T. (2023), "What a Compute-Centric Framework Says About Takeoff Speeds" (Coefficient Giving, formerly Open Philanthropy; interactive model at takeoffspeeds.com): the quantitative takeoff framework; read the summary, then experiment with the interactive model.
• Davidson, T., Hadshar, R. & MacAskill, W. (2025), "Three Types of Intelligence Explosion" (Forethought): the software / AI-technology / full-stack decomposition.
• Narayanan & Kapoor, "AI as Normal Technology", Part I (Knight First Amendment Institute; mandatory sections assigned in Session 1.3): the sceptical counterpart to the takeoff readings above. "Benchmarks do not measure real-world utility" and "Speed limits to progress in AI methods" argue that diffusion and validity constraints, not capability, set the pace: read them against the Anthropic and METR numbers and decide which framing the evidence favours.
• Epoch AI, GATE (epoch.ai/gate): an interactive integrated model of compute growth and automation, for experimenting with the assumptions yourself.
The measurable part: compute trends
Before any argument about "the pace of AI", anchor on what is actually measured. The clearest signal is training compute.
Four findings
- Compute has grown far faster than hardware. Sevilla et al. (2022) date a regime change around 2010: before it, training compute for notable systems roughly tracked Moore's law (a doubling every ~20 months); since then it has doubled roughly every six months, and a separate "large-scale" track of frontier runs sits orders of magnitude above even that. The growth is driven by investment, not cheaper chips alone.
- The three inputs move together. Compute, dataset size and parameters scale in step (2.3), bounded increasingly by data and by energy and hardware supply (Session 4).
- The output side is measured too. METR's time horizon metric asks how long a task (in the time it takes a skilled human) a model can complete with 50% reliability; on their software-task suite that horizon has doubled roughly every seven months since 2019, reaching about 50 minutes for frontier models by early 2025 (Kwa et al., 2025). Capability itself, not only the compute behind it, has a measured trend line.
- The trends are trackable. Groups like Epoch AI maintain public datasets of model compute, parameters and data: the empirical basis for any claim about how fast the field is moving.
You will work with this data directly in the 2.5 lab. When someone says AI is accelerating, or stalling, ask which measured quantity, over what window?
Two implications follow before we even reach the controversy. A six-month doubling means a tenfold jump in training compute is only about twenty months away on the main trend. That lets labs plan frontier runs years ahead: capability is scheduled. And a run at that scale now costs tens to hundreds of millions of dollars, so the frontier concentrates in a handful of well-resourced labs, almost none of them on the African continent. Who can afford the compute is therefore also who sets the defaults the rest of the world inherits, a thread we pick up in compute sovereignty (Session 4) and governance (Session 12).
The contested part: do abilities emerge?
This is a real scientific dispute, and an instructive one, because both sides are looking at the same models.
The claim: emergence is real
Wei et al. (2022), drawing on the BIG-Bench suite, documented emergent abilities: tasks (multi-step arithmetic, word unscrambling, certain reasoning benchmarks) on which performance stays near chance until a scale threshold, then jumps sharply. They define an ability as emergent if it is absent in smaller models and cannot be extrapolated from their scaling curve. If real, this matters for safety: a dangerous capability could appear without warning as models scale.
The rebuttal: a mirage?
Schaeffer, Miranda & Koyejo (2023; NeurIPS Outstanding Paper Award) argued the sharpness is often manufactured by the metric. Score the same runs with a harsh all-or-nothing measure and you see a jump; score them with a smooth measure and the same data show gradual, predictable improvement. On this view much of "emergence" is in the ruler, not the model.
The tasks Wei et al. flagged run from the harmless to the pointed: three-digit addition, transliteration, unscrambling words, multi-step reasoning, and following chain-of-thought prompts that smaller models ignore. For safety the worry is not arithmetic but the same pattern in capabilities we would rather catch early: writing working code, using external tools, persuading a person, or recognising that it is being tested. If those follow the arithmetic curve, they could sit near-absent in every model anyone has checked and present in the next one.
A worked example: exact-match scoring
Take a task whose answer is a 5-token string, scored by exact match (all five tokens right, or zero credit). Suppose the model's per-token accuracy \(p\) improves smoothly with scale: \(p = 0.3, 0.5, 0.7, 0.9\) across four model sizes. If token errors are roughly independent, the exact-match score is \(p^5\):
The underlying skill rose in even steps, but the reported score crawls along near zero and then leaps: an "emergence" curve produced entirely by raising a smoothly-improving quantity to a power. Schaeffer et al. show the converse too: replace exact match with a continuous score (token edit distance, or the log-probability of the correct answer) and the cliff turns into a ramp. Same models, different ruler, opposite story.
The two layers of the dispute
The synthesis is that both readings hold, at different layers. The underlying quantity (loss, or per-token accuracy) improves smoothly and fairly predictably, as 2.3 led us to expect. The user-facing ability, the thing you actually deploy, is frequently a thresholded pass-or-fail quantity, and those can flip sharply even when what drives them moves smoothly. Knowing that a jump is "only" a metric artefact does not help you if the metric is the capability you care about.
Predictable in the aggregate, surprising in the specific
Ganguli et al. (2022) name the structural tension: scaling makes the aggregate loss highly predictable, while leaving specific capabilities (which task switches on at which scale) surprising. You can forecast the curve without being able to forecast what a model on that curve will newly be able to do. For a safety field that is the worst combination: the quantity we can predict is the one we worry about least, and the behaviours we worry about most arrive on a schedule we cannot read in advance.
So the dispute is not a tempest in a benchmark. It tells you which forecasting tools to trust. Smooth loss curves are reliable for budgeting and for predicting loss; they are weak instruments for answering "at what scale does this system become able to design a working exploit, or reliably deceive a grader?" That second kind of question is thresholded, deployment-relevant, and where extrapolation is least safe.
The track record supports caution. Few forecasters called in-context learning in advance, or the degree to which chain-of-thought prompting would unlock multi-step reasoning; both were noticed after they appeared, not predicted before. That is the predictability-and-surprise pattern in practice: the aggregate curve ran on schedule while the specific new behaviours were not anticipated. None of this licenses panic, but it argues against leaning on extrapolation as a safety guarantee.
Recursive self-improvement and takeoff
The strongest version of the pace question is a feedback loop: what happens when AI systems do the work of improving AI?
The idea is old (I. J. Good named the "intelligence explosion" in 1965), but it now has quantitative treatments you can inspect and argue with. Davidson (2023) builds a compute-centric model in which AI progress is driven by compute and algorithmic R&D, and asks what happens as AI itself automates that R&D. In the framework's median run, once AI can automate about 20% of cognitive tasks (weighted by economic value), full automation follows in roughly three years, with clearly superhuman systems shortly after. Takeoff speed names the disputed quantity: how fast capability grows once the loop closes, from "fast takeoff" (months) to "slow takeoff" (decades). The answer hangs on assumptions about returns to software R&D and compute bottlenecks, and you can vary those yourself in the model's interactive version.
The loop is no longer purely hypothetical. Anthropic reported in 2026 that Claude writes over 80% of the company's production code and that the task horizons it handles reliably have been doubling roughly every four months, faster than METR's earlier seven-month estimate (Favaro & Clark, 2026). That is a frontier lab's self-report, so apply the Session 1.4 checklist to it; but it is also the first measured evidence on the input side of Davidson's model rather than an assumption. Davidson, Hadshar & MacAskill (2025) separate three feedback loops that could each sustain an explosion: a software explosion (AI improves algorithms alone), an AI-technology explosion (adding chip design), and a full-stack explosion (adding chip production). They estimate the three could multiply effective compute by roughly 12, 18 and 23 orders of magnitude respectively before hitting physical limits. The structural point matters more than any single number: even if the software loop alone proves insufficient, the slower loops that run through hardware could still produce acceleration, on longer timescales and with more visible industrial footprints.
Takeoff and the forecasting asymmetry
These are models, not measurements, and this sub-session's discipline applies to them directly: ask which input assumptions drive the conclusion, and what near-term evidence would move them. Recursive self-improvement is also where the emergence dispute bites hardest, because "can the model automate AI research?" is exactly the kind of thresholded, deployment-relevant capability that aggregate loss curves are weakest at predicting. That is why AI-R&D automation now appears as a standard threat model in pre-deployment capability evaluations (Session 11), and why compute, the one input you can meter, anchors the governance levers of Sessions 4 and 12.
The calibrated take
Aggregate progress is fast and, in loss terms, fairly predictable. Whether a specific capability arrives smoothly or sharply is unsettled and metric-dependent. So confident predictions of "imminent superintelligence" and of "it has plateaued" are both overclaiming from the same thin evidence. The prudent posture is identical under either reading: build the ability to measure capabilities and detect dangerous ones directly, before deployment (Session 11), rather than trusting any extrapolation to tell you when a threshold will be crossed.
Whose ruler, whose language
Every emergence curve is measured on some benchmark, and almost all of the standard ones are English. That has two consequences for African-language safety. A useful capability can look "switched on" when scored in English while remaining well below threshold in isiZulu or Amharic, because the data that drives the underlying \(p\) is far thinner there (2.3). And a risk (a jailbreak that works, a harmful instruction followed) can be invisible to an English-only evaluation while being live in another language. Which ruler you pick, and in which language, decides what capabilities and what dangers you are even able to see. We make this the heart of the evaluations session (Session 11).
Questions to bring to class
- In the worked example, at what per-token accuracy \(p\) does exact-match cross 0.5 for a 5-token answer? How does that "threshold" move if the answer is 10 tokens long?
- Schaeffer et al. can make emergence appear or vanish by choosing the metric. Does that settle the safety question, or just relocate it? Argue both ways.
- Ganguli et al. say loss is predictable but capabilities are surprising. Which specific capability would you most want an early-warning signal for, and what would you measure to get one?
- If your only evaluations are in English, what kinds of capability and what kinds of risk are you structurally unable to detect?
- Davidson's framework moves from 20% to full automation of cognitive work in about three years. Which input assumption would you attack first, and what evidence available within a year could raise or lower your credence in it?
Next
Time to touch the data yourself. Sub-session 2.5 is the term's first lab: fit a scaling law, explore the compute trends, and open up a real transformer in TransformerLens.