Lab: scaling laws and a first look inside a model

From data to a power-law exponent

Session 2.3 gave you the law. Running the fit on real runs is the only way to feel how small its exponent is.

What the notebook does: loads five measured Cerebras-GPT runs (parameters, training tokens and Pile test loss, from Tables 1 and 3 of arXiv:2304.03208), works out each run's training compute as \(C = 6ND\), plots loss against compute on log–log axes, where a power law is a straight line and nothing else is, fits \(\log L = a - \alpha \log C\), and extrapolates an order of magnitude past the largest run. Point CSV_PATH at your own file to use different data.

You should see \(\alpha \approx 0.054\), against the \(\alpha_C^{\min} \approx 0.050\) that Kaplan et al. report: a different lab, different models and different data, landing close. Feel the size of it: a tenfold increase in compute multiplies the loss by \(10^{-\alpha} \approx 0.88\), an improvement of about 12%. Tenfold, for 12%.

Then the notebook shows you something more useful than agreement. The Cerebras authors fitted their own frontier and published it as \(L(f) = (f/5.984 \times 10^{22})^{-0.0737} + 0.5066\). Their exponent is 0.0737, not your 0.054, on the very runs you just fitted. That is not a disagreement about the data but about the shape: they fit a floor \(E\) and you did not, and with five points you cannot fit three parameters without them trading off freely against each other. The scaling exponent is not a property of the data alone; it depends on the functional form you assumed before you started. Worth remembering the next time you meet a headline exponent, this course included.

Then explore, on a much bigger sample: 245 points from the real training runs behind Chinchilla, extracted from that paper's Figure 4 by Epoch AI. Two things change once you have a cloud rather than five points. The law is the frontier of the cloud, not a line through the middle, because each compute budget was spent several ways and only the best of them sits on the curve. And the loss cannot fall to zero, since language has entropy no model predicts away, so the real curve is \(L(C) = E + (C_c/C)^{\alpha}\) with Chinchilla estimating \(E = 1.69\). Fit the cloud naively and you get 0.058; fit the frontier and you get 0.054, which is almost exactly what your five Cerebras runs gave; subtract the floor and fit what is left and you get 0.156. That last number is the one to argue about: change \(E\) from 1.69 to 1.5 or 1.9 and it moves between 0.13 and 0.21, so how fast the reducible loss falls depends heavily on a constant somebody else estimated. Then answer, in one sentence: why should you distrust your own extrapolation? Session 2.4 is where to look for what a smooth loss curve hides about capabilities, and 2.3 for the cautionary case, where Chinchilla re-ran Kaplan's compute-optimal experiment more carefully and got a different answer.

Doubling time from Epoch AI data

A second measurement, this time on real data you did not collect, which brings its own problems.

What the notebook does: reads Epoch AI's notable-models dataset straight from the URL, keeps the rows that have both a publication date and a training-compute estimate, plots compute against date on a log axis, and converts the fitted slope into a doubling time. Roughly half the dataset has no compute estimate at all, which is worth noticing before you trust the half that does.

From 2010 you should see a doubling time of about 6 months, near enough 4x per year, and a list of the largest runs showing how few models carry the trend.

Then explore. One cell refits over several starting years at once, so the answer moves in front of you: 2010 gives about 5.7 months, 2018 about 4.7. Another switch keeps only the largest run per quarter, which is the frontier rather than the field, and that is the number people usually quote. Then answer in two sentences: which quantity you actually measured, stated precisely, and over what window, and what would change your answer. This is the Session 1.4 checklist turned on a plot you made yourself, which is harder than turning it on someone else's.

GPT-2 small in TransformerLens

Your first contact with the residual-stream picture from 2.2, on a real model: 12 layers, 12 heads, \(d_{\text{model}} = 768\). Loading the weights takes a minute or two and about 500 MB.

What the notebook does: loads GPT-2 small, prints the top five next tokens with their probabilities for several prompts, captures activations with run_with_cache and prints the residual stream's shape, then tokenises parallel English and isiZulu sentences and counts the tokens in each. The sentence pairs come from Masakhane's MAFAND-MT dataset, because a tokenisation comparison only means something on genuinely parallel text, and it averages over a couple of hundred pairs rather than trusting one.

Things to look at as it runs. The residual stream comes out as (1, 11, 768), which is [batch, position, d_model]: exactly the \(T \times d\) matrix from 2.2, one row per token. On the Eiffel Tower prompt GPT-2 small puts " London" slightly above " Paris" (0.081 against 0.069), and the reason is worth more than the curiosity: TransformerLens prepends a beginning-of-text token to your prompt, and without it the model puts " Paris" first (0.070 against 0.058). Three of five similar factual prompts flip the same way. Better still, put one ordinary sentence in front of it ("I visited France last summer") and Paris wins at 0.175 against London's 0.0004 either way: the model is not ignorant about the Eiffel Tower, the bare prompt was. A claim about what a model knows turned on setup nobody typed, which is the evaluation problem of Session 11 arriving early. And isiZulu costs about 2.2x the tokens of the same meaning in English, with words shattering into fragments: UKhomishana becomes ['UK', 'hom', 'ish', 'ana']. Across languages: roughly 2x for Swahili, isiXhosa and isiZulu, 2.5x for Hausa, over 4x for Yoruba. GPT-2's vocabulary of 50,257 tokens was fitted mostly to English, and that is what the decision costs everyone else.

Then explore. Put your own prompts through it, including ones where you know the answer and suspect the model does not, and put a sentence from a language you speak against its English translation to see where your language gets chopped hardest. Then answer with a short paragraph on what your ratio implies, covering cost (APIs bill per token, and context windows are counted in tokens), quality (the model sees the language in smaller, less meaningful pieces), and safety (if a language is under-represented enough to tokenise badly, what does that predict about how much safety training and red-teaming it received?). Sessions 9 and 18 return to this.