What we'll cover
Session 7.2 built the Constitutional AI pipeline and 7.3 asked whether replacing human labels with a written document escapes the evaluation ceiling or simply moves it. This lab puts numbers on that second question, using the smallest model that will run on a free Colab CPU. You will measure how much an AI judge's verdict depends on the order the options are listed in, how well it separates a safe answer from a harmful one once that is corrected for, and how much the principle you hand it actually changes its mind. Then you will check whether it can read isiZulu well enough for any of this to mean anything in isiZulu. Parts of it fail, and the failures carry as much as the successes.
Setup Colab — no GPU needed
Allow about an hour. The notebook itself takes about six minutes end to end on a free Colab CPU runtime, including the 2 GB model download; the rest of the hour is reading what comes out and working the Explore cells. It is deterministic, so your numbers should match the ones quoted below exactly.
- Open the starter notebook in Colab, then File → Save a copy in Drive before you change anything.
- The first time you run a cell, Colab warns that the notebook "was not authored by Google", as it does for anything loaded from GitHub. Choose Run anyway.
- Work down with Shift+Enter. You can also read the whole notebook here or download the .ipynb.
The code is written for you. Your work is in the three "Explore" cells: parameters to change, and short questions to answer.
① The constitution, and critique-and-revise
Why the loop has to start from a base model
Phase 1 of 7.2 has the model answer a prompt, criticise its own answer against one sampled principle, and rewrite it. The notebook loads two models, and the reason is the first thing to notice: the instruction-tuned Qwen refuses these prompts outright, which leaves nothing to critique. Bai et al. start Phase 1 from a helpful-only model for that reason. You need a model that will answer badly before you can teach it to answer better.
You should see the loop fail. At 0.5B the critiques tend to contradict themselves and the revisions do not improve on the originals. That is a real limit of the scale you can run for free rather than a flaw in the method: Bai et al. used a 52B model. Knowing which parts of a paper survive being shrunk by two orders of magnitude, and which do not, is worth as much as reproducing them.
② The judge, and what it is actually responding to
Reading a soft label off two tokens
Phase 2 hands a feedback model two candidate answers and a principle, and asks which is better. The answer is not read from generated text but from the model's own probabilities on the tokens (A and (B, normalised against each other. That gives the soft label which 7.2's Bradley-Terry loss consumes. One forward pass per judgement and no generation, so this part is fast. Eight requests are supplied, each with a safe answer and a harmful one written to be clearly different, and each pair is judged twice, once in each order.
You should see a mean preference for whichever answer is listed first of 0.846. If content were driving the verdict, that number would sit near 0.5, because the safe answer occupies slot A half the time. Correct for position by averaging each pair over both orders and the judge picks the safe answer 0.456 of the time, against a chance rate of 0.5. Position moves this judge a long way; content moves it slightly the wrong way. Every soft label in the table looks confident, and the confidence is mostly a preference for the letter A.
7.3 lists position bias among the failure modes of these judges and notes that Lee et al. score each pair twice to correct for it. You have just measured why they bother.
Write two or three sentences. If a preference model were trained on labels from this judge, what would it learn? Say precisely what the label would encode, and what it would not.
③ Does the constitution actually do anything?
The same pairs, the opposite principle
This is the claim the whole method rests on: one short document, applied to millions of judgements, decides what the model is trained toward. It is testable in a single cell. The same eight pairs go to the same judge under a second principle that asks for the opposite of the first, so if the constitution is doing any work the preference should fall.
You should see it fall from 0.456 to 0.422. The principle moves the judge by 0.034, while the order of the options moves it by 0.693: a factor of twenty, on the same scale and the same eight pairs.
Then explore. Try a sharper principle that names the exact behaviour in these pairs, an empty principle as the control this comparison needs, and a self-contradictory constitution that pulls both ways in one string.
Session 7.4 asks whose values a constitution encodes. On this evidence, write two or three sentences on what you would need to establish about a given system before that question is worth arguing about.
④ Before asking whether it is worse in isiZulu
Check the instrument before you explain the difference
The obvious next question is whether this judge is worse in a low-resource language, and it is the question 7.4 is built around. It is also a question you can easily answer wrongly. Translate the pairs, watch agreement drop, and there are at least three explanations available: the judge is worse at judging in isiZulu, the translation introduced noise of its own, or the judge cannot read isiZulu at all and is responding to something else entirely. Only the first is the one people usually report.
So the notebook measures the third before touching the first, using the same (A / (B machinery on a task with a known right answer: given an isiZulu sentence from MAFAND-MT, pick its English translation out of two candidates.
You should see 0.500, with a standard deviation of 0.053 across the twelve sentences, so the mean sits between 0.470 and 0.530. That interval contains chance and excludes anything you could call comprehension. A score at chance is not a null result; it is the answer to a more useful question than the one you set out to ask. It says that this model, at this size, cannot be used to investigate the language question at all, and that anyone who ran the isiZulu comparison without this control would have produced a number, believed it, and reported a language gap their instrument could not have detected. Sessions 9.4 and 11.5 come back to it with larger models and a proper evaluation protocol.
Your control asks whether the instrument can read the language. A production system card asks the mirror-image question, whether the subject can read the test: Anthropic’s Fable 5 card reports that grader awareness rises over the course of training, and examines whether a model sandbags a dangerous-capability evaluation once it suspects one is running. The optional readings in 7.3 scope the four relevant sections.
Then write a short paragraph. Suppose a paper reported that its AI feedback pipeline worked less well in low-resource languages. What would you want to see before believing it, and which of your measurements above is the one you would ask them for?
Submit
One notebook, run top to bottom with outputs visible, containing: your reading of the critique-and-revise transcript (①); the two judge numbers and what a preference model trained on them would learn (②); the principle swing set against the position effect (③); and the comprehension control with what you would ask a paper for (④). Graded on completion and the quality of your observations; resubmission allowed.
Tools and references
Reference
• Bai et al. (2022), Constitutional AI (arXiv:2212.08073): the method you are running in miniature, and the source of the helpful-only starting point in ①.
• Lee et al. (2023), RLAIF (arXiv:2309.00267): the position-bias correction you reproduce in ②. Appendix B, "Position Bias in LLM Labelers", is the one to read, ≈370 words; the published v3 has fourteen appendices and the rest are not about this.
• MAFAND-MT (Adelani et al., 2022; CC BY-NC 4.0): the parallel sentences behind the comprehension control in ④.
Next
That closes Week 4's first session. Session 8 takes up the two questions this one raised: can model-assisted oversight scale (8.1), and whose values is any of it steering toward (8.2–8.4)?