Session 5.1 — Pretraining: learning to predict the next token

Week 3 — how an LLM is actually trained

Week 2 told us a transformer is trained by minimising next-token loss, and that the gap between the objective we write and the behaviour we want is where alignment lives. This week follows the real pipeline that turns a pile of text into a deployed assistant (pretraining, supervised fine-tuning, and RLHF in Session 6), asking at each stage: what is being optimised, and what does that incentivise?

This session's sub-sessions: 5.1 pretraining · 5.2 supervised fine-tuning · 5.3 the post-training stack · 5.4 lab: nanochat (Colab).

What we'll cover

Everything a frontier model can do begins with pretraining: one enormous run of self-supervised next-token prediction over web-scale text. This sub-session pins down the objective (it is the cross-entropy from Session 2.1, now at scale), explains why it is called "self-supervised", describes what such a run produces (a base model), and why that base model, for all its capability, is not yet a safe, instruction-following assistant. That gap is what Sessions 5.2–6 exist to close.

Mandatory readings

• Sutton, R. (2019), "The Bitter Lesson" (essay, incompleteideas.net): one page, and the argument the rest of the session rests on: general methods that scale beat hand-built structure. Read the whole thing. ≈1,100 words.

• Brown et al. (2020), "Language Models are Few-Shot Learners" (GPT-3; NeurIPS 2020; arXiv:2005.14165): read §1 (Introduction), which sets out in-context learning and why scale changed what a language model is for. The results sections are a reference, not a read. ≈2,000 words.

Total mandatory load: ≈3,100 words.

Optional readings

• Radford et al. (2019), "Language Models are Unsupervised Multitask Learners" (GPT-2; OpenAI technical report): the base-model-as-multitask-learner framing, one generation earlier.

• Brown et al. (2020) (arXiv:2005.14165), §3 onwards: the benchmark results, if you want to see how few-shot performance was actually measured.

The objective: self-supervised next-token prediction

There is no human labelling here. The text labels itself: for every position, the "correct answer" is the token that came next.

Take a corpus of text, tokenise it (Session 2.2), and slide over it. At each position \(i\) the model sees the tokens so far, \(t_{<i}\), and outputs a distribution \(p_\theta(\cdot \mid t_{<i})\) over the vocabulary; the training signal is how much probability it placed on the token \(t_i\) that followed. Averaged over the corpus, the loss is exactly the cross-entropy from Session 2.1:

\[ L(\theta) = -\frac{1}{T} \sum_{i} \log p_\theta\!\left(t_i \mid t_{<i}\right). \]

Why "self-supervised"?

Supervised learning needs labelled examples; producing labels is expensive, which caps dataset size. Next-token prediction sidesteps this: the label for position \(i\) is just \(t_i\), already present in the raw text. So any text is training data, and the supply is effectively the internet. This is the trick that let language models scale: the bottleneck moved from "labelled data" to "compute and raw text".

This is a humble-sounding objective with a profound consequence. To predict the next token well across the breadth of human text (code, dialogue, proofs, translations, arguments), a model is pressured to build internal machinery that captures syntax, facts, reasoning patterns, and styles. Capabilities are not programmed in; they are whatever turns out to reduce next-token loss on a sufficiently varied corpus. This is the seed of the alignment problem: the model learns whatever serves the objective, which is only a proxy for what we want.

The product of pretraining: a base model

The output of a pretraining run is a base model (or "foundation model"): a powerful next-token predictor, and nothing more specific than that.

A base model is, by construction, a model of the training distribution. Prompt it and it continues the text in the most plausible way given what it has read. That makes it astonishingly capable and, simultaneously, not what most users want:

It completes text

Ask a base model "What is the capital of France?" and it might continue with a list of more quiz questions, because on the web, that string is often followed by more questions. It imitates plausible text; it has no notion of "doing what the user asked".

It learns in context

Brown et al. (2020) showed GPT-3 (175B) can perform tasks few-shot, given a handful of examples in the prompt, with no gradient updates. The ability to pick up a pattern from the prompt is itself a learned product of pretraining ("in-context learning").

It contains multitudes

Having modelled the whole web, a base model can imitate the helpful expert and the scammer, the careful reasoner and the conspiracy theorist. It has no fixed persona, only a distribution over all the voices in its data.

Capability without alignment

This is the framing for the rest of the week. Pretraining buys capability (knowledge, reasoning, language) but supplies no alignment: no reliable instruction-following, no stable values, no guardrails beyond whatever correlations exist in the data. Everything we call "the assistant" (helpfulness, honesty, harmlessness, the refusal reflex) is added afterwards, by the much smaller post-training stages (5.2 onward). Alignment is a thin layer applied on top of a vast, indifferent predictor.

Scale and the "bitter lesson"

Why are base models so big? Because, empirically, scale works, and it works predictably (Session 2.3).

The trajectory is stark: GPT-2 (Radford et al., 2019) was a 1.5-billion-parameter model trained on ~40 GB of web text; a year later GPT-3 (Brown et al., 2020) was 175 billion parameters, and the qualitative leap in capability came largely from scale. GPT-2's own paper already noted that zero-shot performance improved "in a log-linear fashion" with model capacity, the scaling-law story you met in 2.3. Richard Sutton's "Bitter Lesson" (2019) names the pattern bluntly: across seventy years of AI, general methods that use more computation tend to win over cleverly hand-engineered ones. Pretraining is that lesson incarnate: a simple objective, applied at enormous scale.

The safety consequences of scale

Two consequences follow directly. First, capabilities (including dangerous ones) arrive as a by-product of scaling an objective that knows nothing of safety; they are present in the base model before anyone decides whether to expose them. Second, because the base model has modelled the whole internet, the safety-relevant question shifts from "can it do X?" (often yes, latently) to "can the thin alignment layer reliably suppress X across all the conditions of deployment?", which, as the isiZulu jailbreak (Session 1.4) showed, it frequently cannot.

The pretraining corpus

If the objective is "predict the next token", then the corpus is everything: it is simultaneously the model's source of capability, of bias, and of latent danger. Pretraining data deserves the same scrutiny as the architecture.

Frontier corpora are assembled from web crawls, books, code, and curated collections, then heavily processed: de-duplication (repeated text distorts what the model "believes" is likely), quality filtering, and removal of the most toxic material. None of this is neutral. Every choice (which languages, which sources, what counts as "low quality") shapes what the model is good at and whose world it represents. Three consequences matter for safety:

The data is the values

A base model has no values we designed; it has the statistical shadow of its corpus. Over-represent one register, dialect or worldview and the model treats it as the default. The web's heavy skew toward English and the Global North is therefore baked in before any alignment, the deepest root of the multilingual safety gap (Session 1.4).

Data is an attack surface

Because the model imitates its data, whoever can place text in the corpus can, in principle, shape behaviour: data poisoning. A handful of crafted documents can plant associations or backdoors that survive into the deployed model (poisoning and its defences are Session 9.5).

High-quality text is finite

The Chinchilla result (Session 2.3) made data a first-class scaling resource, and high-quality human text is a bounded supply. This pressure toward synthetic and lower-quality data is itself a safety variable: the corpus is getting harder to vet just as it matters more.

Predicting the training distribution

This underlies the alignment problem: a base model is optimised to reproduce the distribution of its training data, not to be truthful, helpful, or safe. When those happen to coincide on the data, behaviour looks aligned; where they diverge (rare languages, novel situations, adversarial prompts), the model reverts to "what text is plausible here", which need not be what we want. Pretraining gives us a mirror of the internet, not an agent that shares our goals.

Questions to bring to class

"Self-supervised" learning uses no human labels, yet the result reflects countless human choices. Where do those choices enter?
A base model "contains multitudes": every persona on the web. What does that imply for the claim that a safety-trained model is "safe"? Safe in what sense, and where might the claim break?
If high-quality text is finite and labs turn to synthetic data, what new failure modes might that introduce? (Hint: a model trained partly on its own outputs.)
Why is "the model can do X (latently) but is trained not to" a more accurate (and more worrying) description than "the model can't do X"?

A base model completes text but won't reliably follow instructions. Sub-session 5.2 covers the first fix (supervised fine-tuning) and what it can't do, which is why Session 6 reaches for reinforcement learning.