Note: This page's design and content were created and enhanced using Claude (Anthropic's AI assistant); the reading list has been reference-checked. Spot an error? Email jonathan.shock@uct.ac.za.
Week 4 • Session 7.3

RLAIF and self-rewarding

Generalising beyond Constitutional AI, and the failure modes of automated feedback

What we'll cover

Constitutional AI is one instance of a broader move: replace the human labeller with a model. This sub-session generalises it to plain RLAIF, looks at the evidence that AI feedback can match human feedback (Lee et al.), and at self-rewarding models that judge themselves (Yuan et al.). Then it turns critical. Automating the feedback adds failure modes of its own (self-preference and the risk of a closed data loop), carries over two that human feedback already had (sycophancy and over-optimising a learned proxy) in a form that is harder to check, and forces the session's central question: does AI feedback escape the oversight limitations of 6.4, or move them somewhere harder to see?

Plain RLAIF

RLAIF is the RLHF pipeline of Session 6 with one substitution: an LLM judge in place of the human labeller.

Strip Constitutional AI of its self-critique phase and you are left with the general recipe. Sample response pairs, have a model rank them, train a reward model on those rankings, optimise the policy against it. The constitution, if present, is the rubric handed to the judge; without one, the judge is prompted with a task description ("which response is more helpful and honest?"). The two refinements from 7.2 are both in play, and one is used here but not in Constitutional AI: chain-of-thought reasoning before the judgement, Bai et al.'s device, raises agreement with humans; and scoring each pair in both orders to counterbalance presentation order is a fix Lee et al. (2023, next section) used. One variant drops the separate reward model entirely: direct RLAIF asks the LLM judge for a score on every response during RL and uses that as the reward, trading the cost of querying a large model in the loop for freedom from a reward model that can go stale.

Does AI feedback match human feedback?

Lee et al. (2023) ran the controlled comparison that Constitutional AI did not: RLAIF against RLHF with the rest of the training pipeline held fixed. The answer is broadly yes.

On three tasks (summarisation, helpful dialogue, harmless dialogue) they compared RLAIF against RLHF with human evaluators judging the output. RLAIF and RLHF were preferred over the supervised baseline at statistically indistinguishable rates (71% versus 73% on summarisation, 63% versus 64% on helpful dialogue). On harmlessness, RLAIF came out ahead, scoring 88% harmless against RLHF's 76%. Head-to-head, humans preferred the two equally. For a method that removes human labelling from the loop, "as good as, sometimes better" is a strong result.

AI agreement and the human noise floor

Why should a model's preferences be any good? Because on these tasks the judge already agrees with humans about as often as humans agree with each other. Lee et al.'s best AI labeller reached roughly 78% agreement with the human-preferred option on summarisation, and human inter-annotator agreement on the same data is about 73 to 77%. Once the AI judge's agreement with humans is about as high as humans' agreement with each other, asking it to be "more accurate" is asking it to beat a target that humans themselves cannot agree on. That is what "AI feedback matches human feedback" comes to here: on these tasks the human signal was never clean enough for "more accurate" to mean much.

The result that rules out the easy explanation is the same-size one. You might think RLAIF only works because a big, capable labeller is distilling its knowledge into a smaller policy. So Lee et al. ran it with the labeller the same size as the policy: it still improved over the baseline (68% preferred over the supervised model). They pushed further to a case where the labeller and the initial policy were the exact same checkpoint (a model judging responses from itself) and still saw gains. That rules out distillation from a stronger teacher: the model is bootstrapping on its own judgement, which sets up the next idea.

Self-rewarding: the model as its own teacher

If a model can judge its own outputs, why not let it generate its own training signal and iterate?

Yuan et al. (2024) use one model to generate responses and judge them. High- and low-scored responses become DPO preference pairs, and the process repeats. Their reported improvements over three iterations motivate testing whether feedback quality can improve alongside response quality. They do not establish a universal ceiling on human-trained reward models or guarantee indefinite improvement through self-evaluation.

The authors' own cautions

The paper is careful, and you should be too. The gains were shown over three iterations in one setting; the authors expect the effect to saturate, and flag that response length grew across iterations, while longer answers are known to inflate both human and AI preference scores, so some of the "improvement" may be the judge rewarding verbosity. A model that improves by judging itself is also a model with no external check on the direction it improves in. The mechanism that makes it powerful (closing the loop between generating and judging) is the mechanism the rest of this page worries about.

The failure modes of AI feedback

Removing the human removes a check. Four failure modes follow.

Self-preference bias

An LLM judge tends to over-rate its own outputs. The important subtlety, from Wataoka et al. (2024): the driver is not self-recognition. The judge prefers text with low perplexity (text it finds familiar, that it would have been likely to write) regardless of whether it generated it. Because a model's own outputs are the low-perplexity texts, self-preference falls out as a side effect. GPT-4 showed the strongest effect of the models tested. The consequence for RLAIF: a self-judging loop systematically rewards its own style, narrowing the model toward whatever it already finds fluent.

Inherited and amplified sycophancy

Sharma et al. (2023) found that human preference data favours responses that match the user's stated beliefs, and that humans sometimes prefer a convincing sycophantic response to a correct one (6.4). An AI judge trained to mimic human preferences inherits this bias, and a self-rewarding loop can amplify it: the model learns that agreeable answers score well, judges accordingly, and trains on its own judgement. The preference signal rewards telling people what they want to hear, and automating it does nothing to fix that.

Over-optimising an AI proxy

Gao et al. (2022) measured how policies over-optimise a reward model: push too hard and true quality falls even as the proxy score climbs (6.4). That work used a gold reward model as ground truth. With AI feedback the proxy is itself model-generated, so the gap between proxy and truth can be both larger and less predictable, and you have lost the human signal that would have told you the proxy had drifted. The KL leash of 6.3 still helps, but it is restraining the policy from exploiting an even softer target.

The closed loop: model collapse

Train models on data generated by models, across generations, and quality can degrade: rare patterns vanish and the distribution narrows, a phenomenon Shumailov et al. (2024, Nature) call model collapse. Treat this as an argument by analogy, not a demonstrated property of any single RLAIF run: it is a warning about what closing the data loop tends to do, and self-rewarding training closes that loop. Whether a given scheme collapses depends on how much fresh signal still enters from outside.

Escape, or relocate?

AI feedback changes the information and errors entering training. It may be more reliable than human ratings on some tasks and less reliable on others. A trained model can generalise beyond its feedback, but neither the judge’s competence nor improvement on its scores establishes reliable performance on harder tasks. Evaluate on independent labels and tests, including disagreements between judges. Session 8.1 examines methods for supporting and testing weaker supervisors.

The African lens

The tilt 7.1 flagged can now be given a size. Lee et al.'s labeller agrees with human raters about 78% of the time at its best, and that figure is measured in English; it is not a constant of the method but a property of a model reading the language it knows best. Since AI feedback inherits the base model's competence profile, the cheap and infinitely scalable safety signal now replacing human labour is least trustworthy where human labour is scarcest too. A self-judging loop in isiZulu compounds two weaknesses at once, a weaker generator and a weaker judge, each unable to catch the other. What that costs the people on the other end is 7.4.

Questions to bring to class

Readings

Mandatory readings

• Lee et al. (2023), "RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback" (arXiv:2309.00267): cite the published v3 (ICML 2024); its section numbering differs from the 2023 v1. Read §2.3 for how "AI Labeler Alignment" is defined, §4.1 for the head-to-head, §4.2 for the same-size result, §4.3 for the same-checkpoint result, and §4.4 for the table the alignment figures come from. ≈1,650 words.

• Yuan et al. (2024), "Self-Rewarding Language Models" (arXiv:2401.10020): read §1 for the "superhuman feedback" motivation, then the opening of §2 and §2.4, which states the self-alignment algorithm end to end. ≈1,050 words.

Total mandatory load: ≈2,700 words.

Optional readings

• Lee et al. (v3), §2.1, §2.2.2 and §4.5 (≈1,000 words): how the labelling prompts are built, how direct RLAIF is set up, and the ablation on labeller size. Appendix B is the position-bias study the 7.6 lab reproduces.

• Yuan et al., §2.1–2.3 and §3.2 (≈2,200 words): the components of the loop one by one, and the iteration-by-iteration results.

• Anthropic (2026), "System Card: Claude Fable 5 & Claude Mythos 5" (9 June 2026; anthropic.com/claude-fable-5-mythos-5-system-card, a 317-page PDF): the failure modes argued for on this page, documented instead in a production system. Four scoped sections, ≈4,350 words in all. §6.2.3, pp. 108–109, an automated behavioural audit built from about 2,900 investigation sessions. §6.4.1.2, pp. 164–166, “Unverbalized awareness of being evaluated or graded”. §6.4.2, pp. 170–175, grader awareness, including the finding that it rises over the course of training. §6.5.2, pp. 195–201, potential sandbagging on dangerous-capability evaluations. Cite this 5.0 card specifically: the 5.1 card of 1 September 2026 is a different document of 212 pages, and these section numbers and page ranges do not carry across to it.

• Wataoka et al. (2024), "Self-Preference Bias in LLM-as-a-Judge" (arXiv:2410.21819): the perplexity/familiarity mechanism, with GPT-4 as the strongest case.

• Sharma et al. (2023), "Towards Understanding Sycophancy in Language Models" (arXiv:2310.13548): preference data rewards agreement over correctness (recap from 6.4).

• Gao et al. (2022), "Scaling Laws for Reward Model Overoptimization" (arXiv:2210.10760): over-optimisation, now of an AI proxy (recap from 6.4).

• Shumailov et al. (2024), "AI models collapse when trained on recursively generated data" (Nature 631, 755–759; preprint arXiv:2305.17493): model collapse, used here as argument by analogy.

Next

Sub-session 7.4 takes up the normative question the whole session has been deferring: whose constitution? Democratic value-setting, the African-language evaluation gap, and the warning against tokenistic participation.