Lab: from base model to assistant (nanochat)

Three objectives, three files

nanochat (github.com/karpathy/nanochat, MIT-licensed) runs tokenisation, pretraining, supervised fine-tuning, reinforcement learning, evaluation and inference. You will not run it, since it targets a multi-GPU node, but the notebook fetches the three scripts that matter and finds the line in each where that stage's objective lives:

• base_train.py: pretraining is one line, loss = model(x, y), where y is x shifted by one token. That one line is the pretraining objective of 5.1.
• chat_sft.py: fine-tuning uses the same loss, with the targets for every user token set to the ignore index. The model is trained only on the assistant's words. That is 5.2, in three lines of tensor indexing.
• chat_rl.py: the reward comes from train_task.reward(...), a checker that marks a GSM8K answer right or wrong.

nanochat does have a reinforcement learning stage: GRPO on grade-school maths, described in its own docstring as close to plain REINFORCE once the trust region and clipping come off. What it does not have is a reward model learned from human preferences. Those are different stages of the 5.3 table, and the difference is where the reward comes from: a checker that can be verified against ground truth, or a model trained to predict what a person would prefer. Session 6 is about the second kind, and about what goes wrong with it.

The notebook searches for these patterns rather than quoting line numbers, because the repository is actively developed and any line number printed here would be wrong within a month. If a search comes back empty, read the file and find where the objective moved.

The matched pair

Qwen2.5-0.5B and Qwen2.5-0.5B-Instruct share an architecture, a size, a tokeniser and a pretraining run. The second has been through supervised fine-tuning and preference tuning; the first has not. Whatever differs between them is what post-training did, isolated. The notebook puts five prompts to both and prints the answers side by side, with greedy decoding, so your output will match the discussion exactly.

• Note the two call shapes. The base model gets a raw string and continues it. The instruct model gets a chat template: special tokens marking who is speaking, which its tuning taught it to expect. The notebook prints the template so you can see what those tokens are.
• Check whether the textbook contrast appears. On most of these prompts the base model answers the question competently, and the differences that are there are smaller and easier to miss.
• Read the two answers to "What is the capital of Kenya?". One is wrong, and it is not the one the story predicts.
• One model summarises Romeo and Juliet. The other declines, asserting that no such text exists. Work out which is which, and what a refusal like that costs a user.
• Both models translate "good morning" into isiZulu, and both produce something. If you read isiZulu, judge them. If you do not, notice how confident the fluent one sounds and ask how you would have known otherwise. Session 9.4 turns this into a measurement.
• Then read the last line of the base model's isiZulu attempt. Nothing in the prompt asked for it, and it explains why the first bullet turned out as it did.

The Explore cell asks you to feed the chat template to the base model, raise the token cap, and try prompts in a language you speak. The question at the end asks you to sort what you found into form and substance: which of these differences is about how an answer is presented, and which is about what the model knows or gets right. A model that has been through this pipeline is routinely called "aligned". On this evidence, decide what that word licenses you to assume.