Note: This page's design and content were created and enhanced using Claude (Anthropic's AI assistant); the reading list has been reference-checked. Spot an error? Email jonathan.shock@uct.ac.za.
Week 4 • Session 7.1

From human to AI feedback

Why the field replaced most of its human raters with a model, and what human input remains

What we'll cover

Human feedback can be costly and unreliable on difficult tasks. AI feedback replaces some human labels with model judgements, sometimes guided by a written constitution. We examine what this automates, what still depends on people and how to evaluate the resulting signal.

The wall at the end of Session 6

Two limits closed in on RLHF at once: an evaluation limit and a throughput limit.

When labellers cannot reliably judge an answer, their preferences can favour fluent errors. More labels may reduce random noise, but do not by themselves remove shared mistakes. A reward model can generalise beyond the labels it receives; whether that generalisation improves judgement needs independent evaluation.

The throughput limit is more mundane and just as binding. Frontier preference datasets run to hundreds of thousands of human comparisons; each one is a person reading two responses and deciding. That is slow, costly, and hard to scale to every new capability, language, and harm you would like to cover. Lee et al. (2023, covered in 7.3) estimate AI preference labelling at more than ten times cheaper than human annotation. It returns labels in seconds, not days. When a method is both cheaper and faster by an order of magnitude, the field moves whether or not the quality question is settled. Part of this session is insisting we keep asking the quality question anyway.

The throughput gap in numbers

Put rough numbers on it. Suppose one comparison takes a human rater 30 seconds of careful reading. A 100,000-comparison preference set is then about 830 hours of labelling: months of one person's full-time work, or a sizeable paid crowd, before a single policy update. An LLM labeller returns the same 100,000 judgements in the time it takes to run the inference, at an estimated tenth of the cost. The pressure behind the pivot is scale and price: the AI labels arrive faster and cheaper than human labelling can manage, whatever their quality. What that speed costs in reliability is the question 7.3 takes up.

The labour behind the number

Those 830 hours are a throughput figure here, and a price. They are also months of somebody's full-time work. Session 6.1 carries a related case: safety labelling is often subcontracted, paid at a fraction of what the contracting lab pays for the time, and some of it, such as red-teaming and toxic-content filtering, uses material chosen for being harmful.

That gives the pivot to AI feedback an argument in its favour which the cost framing does not capture. Fewer people have to read this material at all.

The same framing hides a cost on the other side. The humans being removed are the ones whose judgement was the justification for the method in the first place. "Human feedback" is where the legitimacy came from, so automating it does not only make the pipeline faster, it removes the part that made the result answerable to anyone outside the lab. That worry is separate from the reliability question 7.3 takes up, and it survives even if the AI labels turn out to be every bit as good: the question stops being "are these judgements accurate" and becomes "whose judgements are these". 7.2 and 7.4 are where the course tries to answer it.

Let a model give the feedback

Replace the human labeller with a model that judges responses against a written list of principles.

The idea, introduced by Bai et al. (2022) under the name Constitutional AI, is disarmingly simple. Instead of paying people to rank responses for harmlessness, you write down a handful of principles (choose the less harmful response, choose the less discriminatory one, and so on) and you ask a capable model to do the ranking by those principles. The model's judgements become the preference data; everything downstream (reward model, policy optimisation) is the Session 6 machinery, unchanged. In the reinforcement-learning stage, only the label source changed. Constitutional AI also adds a supervised stage before it, in which the model critiques and revises its own responses (7.2).

It helps to be exact about what is and is not automated, because the marketing and the mechanism diverge here.

Automated and human inputs

  • Automated: the per-comparison judgement. A model decides which of two responses better satisfies a principle, and does so at machine speed and cost.
  • Still human: the principles themselves. Someone writes the constitution and chooses what goes in it. Also still human: the base model's pretraining data, the helpfulness preference data (which in Constitutional AI stays human-labelled), the human-written red-team prompts the method starts from (most of the red-team prompts it trains on are model-generated), and the decision to ship. The human role moves up a level, from labelling instances to writing the rule the labeller applies.

That relocation is the appeal: writing a hundred principles once is far cheaper than collecting a hundred thousand labels, and a principle is auditable in a way a pile of individual judgements is not. It is also the worry, which is why the next claim deserves scrutiny.

"The only human oversight is the constitution"

Bai et al. describe their method as training a harmless assistant "without any human labels identifying harmful outputs", where "the only human oversight is provided through a list of rules or principles". Read literally that is a striking claim, and so hold it up to the light rather than nodding along, in the model-student spirit of Session 1.

Is the constitution the only human input?

No. The authors' phrase is about the harmlessness labels specifically, not the whole pipeline. The base model was pretrained on human text and fine-tuned for helpfulness on human preference data; the critiques and revisions come from a helpful RLHF model trained on human preferences, and the default feedback labeller is a model pretrained on human text; the human-written red-team prompts, the principle wording, and the choice of which principles to include are all human acts. So the constitution is not the only human fingerprint; it is the only place where humans label harmfulness directly. The accurate statement is narrower and more interesting: the method removes humans from the harmfulness labelling loop while concentrating their influence in a small, visible document. Whether that is reassuring or alarming is what 7.4 takes up.

Sharpening the claim this way is not pedantry. If you believe the constitution is the only human input, then "whose values?" reduces to "who wrote these few lines?" (a small, fixable question). Once you see the human fingerprints all through the stack, you realise the constitution is the visible tip of a much larger set of choices, most of them made in a handful of Northern labs. The narrow reading is comforting; the accurate reading is the course's whole argument.

Three questions this session has to answer

Automating the feedback raises a technical question, a normative one and a practical one. The session is organised around them.

Does AI feedback improve the training signal on the tasks where human feedback is unreliable or costly? Replacing the judge changes its strengths and failure modes. Compare its labels against independent evaluations, including examples where it disagrees with humans. Session 8.1 examines methods for supporting limited overseers and testing generalisation beyond their supervision.

The normative question: whose principles? A constitution is a single, explicit value commitment applied to millions of judgements. Writing it down makes the values auditable, which is progress over a reward model's implicit, buried preferences. It also makes one blind spot scale to everyone who uses the model. Who should write it, and what happens to the people that the writers do not represent, is 7.4. That is where this session becomes African technical AI safety rather than a recap of an Anthropic paper.

The practical question: does the document bind anything? A constitution is a piece of writing, and writing only governs a deployed system if the system actually follows it. 7.5 turns to the specifications the labs publish for production models, and to what an independent audit finds when it checks a shipped model against its own stated rules.

The African lens

"Whose principles?" has a default answer that the market has already given: the principles are written in English, by and for a Northern user base, and applied worldwide. The same is true one layer down, of the model that applies them: 7.3 puts a number on that. So both the rule and the judge are tilted before any African user appears, which is why 7.4 is the centre of this session rather than an appendix to it. Session 8.4 supplies the formal argument that there is no neutral way to aggregate the values left out.

Questions to bring to class

Readings

Mandatory readings

• Bai et al. (2022), "Constitutional AI: Harmlessness from AI Feedback" (arXiv:2212.08073). Read the abstract, then §1.2 for the shape of the approach and §1.3 for what the authors claim to have shown. That is where the "only human oversight is a constitution" framing is stated, which is the claim this page spends the rest of its length testing. ≈900 words; the mechanism is 7.2.

Total mandatory load: ≈900 words.

Optional readings

• Bai et al., §1.1 (Motivations): the authors' own case for AI feedback, at ≈1,100 words. It covers the same ground as this page, so read it if you want their framing rather than ours.

• Casper et al. (2023), "Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback" (arXiv:2307.15217): the catalogue of RLHF limits from 6.4. Read §3.1, on the difficulty of getting good human feedback: that is the wall this session climbs over, or relocates. ≈2,400 words. (§3 as a whole runs to about 5,200 and covers the reward model and the policy as well, which 6.4 already dealt with.)

Next

Sub-session 7.2 builds Constitutional AI end to end: the supervised critique-and-revise phase, then RL from AI feedback, with a worked critique example and the helpful-versus-harmless split worked through.