What we'll cover
Session 6 ended at a wall: RLHF can only align a model as far as its human raters can tell good from bad, and those raters are slow and expensive. This sub-session is about the dominant industrial response: have a model supply the feedback, steered by a short written list of principles called a constitution. We set out what gets automated, what stays human, and the three questions the rest of the session has to answer. The mechanism itself is 7.2; here we establish the motivation and sharpen the claims so they can be tested rather than swallowed.
The wall at the end of Session 6
Two limits closed in on RLHF at once: an evaluation limit and a throughput limit.
The evaluation limit is the one we derived in 6.4. A reward model is trained to predict which response a human labeller prefers, so it can only ever encode preferences a human could express from looking at the response. Where the labeller cannot reliably judge (a long proof, a subtle factual error, a security flaw buried in plausible code), the signal is noise, and optimising against noise rewards whatever looks right to a non-expert. That ceiling does not lift by collecting more labels of the same kind. It is a property of who is doing the judging.
The throughput limit is more mundane and just as binding. Frontier preference datasets run to hundreds of thousands of human comparisons; each one is a person reading two responses and deciding. That is slow, costly, and hard to scale to every new capability, language, and harm you would like to cover. Lee et al. estimate AI preference labelling at more than ten times cheaper than human annotation. It returns labels in seconds, not days. When a method is both cheaper and faster by an order of magnitude, the field moves whether or not the quality question is settled. Part of this session is insisting we keep asking the quality question anyway.
The throughput gap in numbers
Put rough numbers on it. Suppose one comparison takes a human rater 30 seconds of careful reading. A 100,000-comparison preference set is then about 830 hours of labelling: months of one person's full-time work, or a sizeable paid crowd, before a single policy update. An LLM labeller returns the same 100,000 judgements in the time it takes to run the inference, at an estimated tenth of the cost. The pressure behind the pivot is scale and price: the AI labels arrive faster and cheaper than human labelling can manage, whatever their quality. What that speed costs in reliability is the question 7.3 takes up.
The labour behind the number
Those 830 hours are a throughput figure here, and a price. They are also somebody's working month. Session 6.1 carries the case: this labelling is often subcontracted, paid at a fraction of what the contracting lab pays for the time, and the material is chosen for being harmful.
That gives the pivot to AI feedback an argument in its favour which the cost framing does not capture. Fewer people have to read this material at all.
The same framing hides a cost on the other side. The humans being removed are the ones whose judgement was the justification for the method in the first place. "Human feedback" is where the legitimacy came from, so automating it does not only make the pipeline faster, it removes the part that made the result answerable to anyone outside the lab. That worry is separate from the reliability question 7.3 takes up, and it survives even if the AI labels turn out to be every bit as good: the question stops being "are these judgements accurate" and becomes "whose judgements are these". 7.2 and 7.4 are where the course tries to answer it.
Let a model give the feedback
Replace the human labeller with a model that judges responses against a written list of principles.
The idea, introduced by Bai et al. (2022) under the name Constitutional AI, is disarmingly simple. Instead of paying people to rank responses for harmlessness, you write down a handful of principles ("choose the response that is least harmful", "that is least likely to be discriminatory", and so on) and you ask a capable model to do the ranking by those principles. The model's judgements become the preference data; everything downstream (reward model, policy optimisation) is the Session 6 machinery, unchanged. Only the label source changed.
It helps to be exact about what is and is not automated, because the marketing and the mechanism diverge here.
Automated and human inputs
- Automated: the per-comparison judgement. A model decides which of two responses better satisfies a principle, and does so at machine speed and cost.
- Still human: the principles themselves. Someone writes the constitution and chooses what goes in it. Also still human: the base model's pretraining data, the helpfulness preference data (which in Constitutional AI stays human-labelled), the red-team prompts the method trains on, and the decision to ship. The human role moves up a level, from labelling instances to writing the rule the labeller applies.
That relocation is the appeal: writing a hundred principles once is far cheaper than collecting a hundred thousand labels, and a principle is auditable in a way a pile of individual judgements is not. It is also the worry, which is why the next claim deserves scrutiny.
"The only human oversight is the constitution"
Bai et al. describe their method as training a harmless assistant "without any human labels identifying harmful outputs", where "the only human oversight is provided through a list of rules or principles". Read literally that is a striking claim, and so hold it up to the light rather than nodding along, in the model-student spirit of Session 1.
Is the constitution the only human input?
No. The authors' phrase is about the harmlessness labels specifically, not the whole pipeline. The base model was pretrained on human text and fine-tuned for helpfulness on human preference data; the feedback model that applies the constitution is itself a product of earlier human alignment; the red-team prompts, the principle wording, and the choice of which principles to include are all human acts. So the constitution is not the only human fingerprint; it is the only place where humans label harmfulness directly. The accurate statement is narrower and more interesting: the method removes humans from the harmfulness labelling loop while concentrating their influence in a small, visible document. Whether that is reassuring or alarming is what 7.4 takes up.
Sharpening the claim this way is not pedantry. If you believe the constitution is the only human input, then "whose values?" reduces to "who wrote these few lines?" (a small, fixable question). Once you see the human fingerprints all through the stack, you realise the constitution is the visible tip of a much larger set of choices, most of them made in a handful of Northern labs. The narrow reading is comforting; the accurate reading is the course's whole argument.
Three questions this session has to answer
Automating the feedback raises a technical question, a normative one and a practical one. The session is organised around them.
The technical question: does AI feedback escape the evaluation ceiling, or merely relocate it? A human-bounded reward model was capped at human judgement. An AI-bounded one is capped at the feedback model's judgement instead. That is certainly a different bound; whether it is a higher one is the thing to settle rather than assume, and 7.3 settles it against the evidence. The menu of methods that try to raise the ceiling outright (debate, amplification, weak-to-strong) is Session 8.1.
The normative question: whose principles? A constitution is a single, explicit value commitment applied to millions of judgements. Writing it down makes the values auditable, which is progress over a reward model's implicit, buried preferences. It also makes one blind spot scale to everyone who uses the model. Who should write it, and what happens to the people that the writers do not represent, is 7.4. That is where this session becomes African technical AI safety rather than a recap of an Anthropic paper.
The practical question: does the document bind anything? A constitution is a piece of writing, and writing only governs a deployed system if the system actually follows it. 7.5 turns to the specifications the labs publish for production models, and to what an independent audit finds when it checks a shipped model against its own stated rules.
The African lens
"Whose principles?" has a default answer that the market has already given: the principles are written in English, by and for a Northern user base, and applied worldwide. The same is true one layer down, of the model that applies them: 7.3 puts a number on that. So both the rule and the judge are tilted before any African user appears, which is why 7.4 is the centre of this session rather than an appendix to it. Session 8.4 supplies the formal argument that there is no neutral way to aggregate the values left out.
Questions to bring to class
- State the evaluation ceiling from 6.4 in one sentence. Why does swapping a human labeller for an AI one not, by itself, lift it?
- "The only human oversight is the constitution." List three human inputs to a Constitutional-AI model that this phrase glosses over. Does naming them weaken the method, or only describe it accurately?
- Writing principles down instead of collecting labels makes values auditable but also makes one choice scale. Which property matters more, and to whom?
- If AI feedback is ten times cheaper and returns in seconds, what would have to be true about its quality for the trade to be a bad one anyway?
Readings
Mandatory readings
• Bai et al. (2022), "Constitutional AI: Harmlessness from AI Feedback" (arXiv:2212.08073). Read the abstract, then §1.2 for the shape of the approach and §1.3 for what the authors claim to have shown. That is where the "only human oversight is a constitution" framing is stated, which is the claim this page spends the rest of its length testing. ≈900 words; the mechanism is 7.2.
Total mandatory load: ≈900 words.
Optional readings
• Bai et al., §1.1 (Motivations): the authors' own case for AI feedback, at ≈1,100 words. It covers the same ground as this page, so read it if you want their framing rather than ours.
• Casper et al. (2023), "Open Problems and Fundamental Limitations of RLHF" (arXiv:2307.15217): the catalogue of RLHF limits from 6.4. Read §3.1, on the difficulty of getting good human feedback: that is the wall this session climbs over, or relocates. ≈2,400 words. (§3 as a whole runs to about 5,200 and covers the reward model and the policy as well, which 6.4 already dealt with.)
Next
Sub-session 7.2 builds Constitutional AI end to end: the supervised critique-and-revise phase, then RL from AI feedback, with a worked critique example and the helpful-versus-harmless split worked through.