Session 2.3 — Neural scaling laws

What we'll cover

One of the most consequential empirical findings in modern AI is also one of the simplest to state: make a transformer bigger, train it on more data with more compute, and its loss falls along a strikingly regular power law. This sub-session makes that precise. We write down the three scaling laws Kaplan et al. found, derive the compute-optimal trade-off that the Chinchilla paper corrected, work a budget example with real numbers, and meet the data wall that now bounds the whole enterprise. Then we draw out the part that matters for this course: capability is on a forecastable trajectory while safety is not, and that asymmetry is the engine behind the alignment problem.

Mandatory readings

• Kaplan, J., et al. (2020), "Scaling Laws for Neural Language Models" (arXiv:2001.08361): read §1 (including §1.2, Summary of Scaling Laws) and the basic power laws in §3. ≈2,500 words.

• Hoffmann, J., et al. (2022), "Training Compute-Optimal Large Language Models" (Chinchilla; arXiv:2203.15556): read §1 and §3 (the three estimation approaches), then skim the Chinchilla results in §4.2. ≈3,000 words.

• Gwern (2020), "The Scaling Hypothesis" (gwern.net): the argument that scale is the main driver of progress, written in 2020 before it was consensus. Read three sections: "Scaling" (the empirical curves; skim, it reinforces Kaplan and Chinchilla), "Scaling Hypothesis" (the thesis, and the author's own update from sceptic to believer), and "Why Does Pretraining Work?" (why falling loss buys capability at all: predicting the hard residual tokens forces reasoning and world knowledge). ≈7,000 words. The rest of the essay (the GPT-3 demos, Blessings of Scale, Prospects) is an optional tail.

Total mandatory load: ≈12,500 words.

Optional readings

• Sevilla, J., et al. (2022), "Compute Trends Across Three Eras of Machine Learning" (arXiv:2202.05924): how fast training compute has grown.

• Villalobos, P., et al. (2022), "Will we run out of data? Limits of LLM scaling based on human-generated data" (arXiv:2211.04325): the data wall.

• Epoch AI: live data on model size, compute and data trends (epoch.ai).

The empirical law

Kaplan et al. (2020) measured the test loss \(L\) of language models across roughly seven orders of magnitude and found it scales as a power law in each of three quantities (model parameters \(N\), dataset size \(D\), and training compute \(C\)) whenever the other two are not the bottleneck.

Written out, the three relationships are

\[ L(N) \approx \left( \frac{N_c}{N} \right)^{\alpha_N}, \qquad L(D) \approx \left( \frac{D_c}{D} \right)^{\alpha_D}, \qquad L(C) \approx \left( \frac{C_c}{C} \right)^{\alpha_C}. \]

Each is a straight line on a log–log plot, with slope equal to minus the exponent. The exponents are small and positive: Kaplan estimated \(\alpha_N \approx 0.076\), \(\alpha_D \approx 0.095\), and \(\alpha_C^{\min} \approx 0.05\) along the compute-efficient frontier (the plain compute exponent is \(\alpha_C \approx 0.057\)). A small exponent means diminishing returns: a \(10\times\) increase in parameters cuts the (reducible) loss by only a factor of about \(10^{0.076} \approx 1.19\). But the line does not bend down to a plateau anywhere in the studied range.

An irreducible floor

Loss cannot fall to zero: natural language has irreducible unpredictability, so there is an entropy floor \(L_\infty\) no model can beat. The complete form of the law separates that floor from the part scale can buy:

\[ L(N) = L_\infty + \left( \frac{N_c}{N} \right)^{\alpha_N}. \]

The power law describes the reducible loss, the gap above the floor. Two facts make the finding remarkable: its regularity (clean power laws over many orders of magnitude) and its smoothness (no plateau in sight). Within wide limits, architectural details (exact depth, width, number of attention heads) move the loss far less than scale does. That is what licenses talking about "scale" as a single dial.

The cost of training

To turn this into a budgeting question we need the cost of training. A useful rule of thumb is that one forward-and-backward pass costs about six floating-point operations per parameter per token (two for the forward pass, roughly four for the backward pass), so total training compute is

\[ C \approx 6\,N\,D \quad \text{(FLOPs)}. \]

The active parameter count

The six in \(6ND\) comes from counting operations per parameter per token, which assumes every parameter is used on every token. In a dense transformer that holds. In a mixture-of-experts model (2.2) it does not: the router sends each token through a small fraction of the feed-forward parameters, so the compute is set by the active parameter count, not the total.

The gap is not a rounding error. Mixtral 8x7B holds 47 billion parameters and uses about 13 billion per token, so putting the total into \(6ND\) overstates its training compute by roughly a factor of three and a half. On models with more experts the factor is larger. Sparsity is standard among the largest open-weight models and widely reported for closed ones, so check which kind of model you are reasoning about before applying the rule.

Two things follow. When you see a compute estimate, check which \(N\) went into it. And a sparse model and a dense model with the same total parameter count are not comparable, on cost or on capability: they differ in what they can store and in what they spend to use it. The Chinchilla trade-off below is still a trade-off, but the axis you are trading along is active parameters against tokens.

The models in the 2.5 lab are all dense, so \(6ND\) is the right rule there.

With \(N\) in parameters and \(D\) in tokens, a fixed budget \(C\) buys a hyperbola of choices: a big model on little data, or a small model on lots of data, or anything between. The natural question is where on that curve the loss is lowest, and for years the field answered it wrong.

Compute-optimal training: Chinchilla

Spend a fixed compute budget well

Hoffmann et al. (2022), the Chinchilla paper, fitted a single parametric law to loss as a joint function of size and data, then asked how to split a fixed budget. Their result: the field had been training models that were too large and too data-starved. For compute-optimal training, parameters and tokens should grow in near-equal proportion, at roughly 20 training tokens per parameter. A smaller model trained on more data beats a much larger model trained on less, at the same compute.

The law they fitted decomposes loss into the entropy floor plus a finite-model penalty plus a finite-data penalty:

\[ L(N, D) = E + \frac{A}{N^{\alpha}} + \frac{B}{D^{\beta}}. \]

The estimated exponents come out close to each other (\(\alpha \approx 0.34\), \(\beta \approx 0.28\)). Minimising \(L\) subject to \(C \approx 6ND\) then implies that the optimal \(N\) and \(D\) both grow as roughly the square root of the budget, so their ratio stays nearly constant as you scale. That constant ratio is the 20-tokens-per-parameter rule.

Chinchilla vs Gopher: the budget arithmetic

At a fixed budget of about \(5.8 \times 10^{23}\) FLOPs, DeepMind had trained Gopher at 280B parameters. Chinchilla spent the same compute on a 70B model trained on 1.4T tokens (four times smaller, four times more data) and beat Gopher across the board. Check the arithmetic with \(C \approx 6ND\): \(6 \times (70\times10^9) \times (1.4\times10^{12}) \approx 5.9\times10^{23}\), and \(1.4\text{T} / 70\text{B} = 20\) tokens per parameter.

Now run it forward. Suppose you hold a budget \(C = 10^{24}\) FLOPs and follow the rule \(D \approx 20N\). Then \(C \approx 6N(20N) = 120\,N^2\), so \(N \approx \sqrt{10^{24}/120} \approx 9\times10^{10}\), about a 90B-parameter model, trained on \(D \approx 1.8\) trillion tokens. The law turns a vague "make it bigger" into a specific, checkable recipe.

The ratio is a rule for minimising loss at a fixed training budget, and production models no longer follow it: once a model will be served at scale, inference cost dominates, so it pays to overtrain a smaller model far past the Chinchilla point. Llama 3 8B, for example, was trained on about 15 trillion tokens, roughly 1,875 per parameter against the rule's 20.

The correction to Kaplan

This is an example of an empirical result being overturned by a better-controlled one. Kaplan et al. (2020) had concluded that model size should grow much faster than data, which is why the models of that era (GPT-3 at 175B parameters on 300B tokens, under two tokens per parameter) were enormous and badly undertrained. Chinchilla traced the discrepancy partly to the learning-rate schedule. Kaplan used a single fixed cosine schedule rather than decaying it to match each shorter training horizon. That inflated the loss estimates for the smaller-data runs, made training on less data look less effective than it is, and pushed the apparent optimum toward over-large, under-trained models. Fix the schedule, and the optimum shifts to equal scaling. For a safety course the record is worth keeping in mind: confident, widely-followed scaling advice was wrong for two years, and only careful re-measurement caught it.

The data wall

If data must scale in step with parameters, then data becomes a binding constraint alongside chips, because high-quality human text is finite. Villalobos et al. (2022) estimate that the usable stock of public, high-quality text could be effectively exhausted within this decade if demand keeps growing at recent rates. That projection drives the current scramble for synthetic data, multi-epoch training, and licensing deals. It is also why "what are we training on?" is now a frontier research question rather than a detail.

African languages and the data term

Read the law again with the data term in view. A model is starved of capability in any language for which the \(B/D^{\beta}\) penalty stays large, that is, wherever high-quality training text is scarce. For most African languages it is very scarce: isiZulu, Yoruba, Amharic and hundreds of others contribute a tiny fraction of the web text these models see. So the weaker performance and thinner safety guarantees in those languages are, in large part, a scaling-inputs gap rather than an architecture gap. "Just scale the model up" does not close it. That is why data sovereignty (Session 4) and in-language evaluation (Session 11) are technical safety questions, not diversity add-ons. The isiZulu-jailbreak result from Session 1 is this law biting.

The forecasting asymmetry

The reason a safety course opens with an empirical curve is that the curve sets the terms of the whole problem.

Capability is forecastable

Loss, and more loosely many capabilities, can be predicted before a model is built. Frontier labs commit budgets to larger systems partly because the payoff is foreseeable. Capability sits on a planned trajectory that you can extrapolate.

Safety is not forecastable the same way

There is no clean scaling law for "will not deceive its overseer" or "is robust to jailbreaks". The quantity we can extrapolate is the one we worry about least; the properties we care about most resist this kind of prediction.

Compute becomes the lever

If capability tracks compute, then compute is the thing you can forecast, meter, and govern (Sessions 4 and 12). Scaling laws are the reason "compute governance" is even a coherent idea.

That asymmetry is the core of the field. We can buy a known reduction in loss with a known quantity of compute and data, and we cannot buy a known increase in honesty or controllability the same way. Progress on the axis we can measure is funded and scheduled; progress on the axes we cannot is neither. Much of the rest of the course is an attempt to build measurement and control for the properties that scaling does not hand us for free.

What scaling laws do not say

They describe smooth falls in aggregate loss. They do not promise that every specific capability appears smoothly as scale grows, nor that loss keeps falling forever, nor that more scale is the only route to capability. The relationship between a smoothly-falling loss and the sometimes-sudden appearance of a specific skill is the controversy we take up next.

Questions to bring to class

The compute-optimal exponents \(\alpha\) and \(\beta\) are nearly equal. Show why that makes the optimal tokens-per-parameter ratio roughly constant as the budget grows.
Kaplan and Chinchilla used the same kind of data and reached opposite advice about how to spend compute. What does that disagreement tell you about how much to trust a single scaling study?
"Capability is forecastable; safety is not." Name one capability you think is on a predictable curve and one safety property you think is not. Say what makes the difference.
If the binding constraint becomes data rather than compute, which governance levers (Session 12) lose their grip, and which become more important?

Loss falls smoothly, but do abilities? Sub-session 2.4 takes on the "emergence" debate: whether new skills appear suddenly at scale or whether that suddenness is an artefact of how we measure, and what each answer would mean for predicting dangerous capabilities.