RLAIF and self-rewarding

Plain RLAIF

RLAIF is the RLHF pipeline of Session 6 with one substitution: an LLM judge in place of the human labeller.

Strip Constitutional AI of its self-critique phase and you are left with the general recipe. Sample response pairs, have a model rank them, train a reward model on those rankings, optimise the policy against it. The constitution, if present, is the rubric handed to the judge; without one, the judge is prompted with a task description ("which response is more helpful and honest?"). The two refinements from 7.2 are both in play, and one originates here rather than in Constitutional AI: chain-of-thought reasoning before the judgement, Bai et al.'s device, raises agreement with humans; and scoring each pair in both orders to cancel position bias is the fix Lee et al. (2023, next section) introduced. One variant drops the separate reward model entirely: direct RLAIF asks the LLM judge for a score on every response during RL and uses that as the reward, trading the cost of querying a large model in the loop for freedom from a reward model that can go stale.

Does AI feedback match human feedback?

Lee et al. (2023) ran the head-to-head that Constitutional AI never did, and the answer is broadly yes.

On three tasks (summarisation, helpful dialogue, harmless dialogue) they compared RLAIF against RLHF with human evaluators judging the output. RLAIF and RLHF were preferred over the supervised baseline at statistically indistinguishable rates (71% vs 73% on summarisation, 63% vs 64% on helpful dialogue). On harmlessness, RLAIF came out ahead, scoring 88% harmless against RLHF's 76%. Head-to-head, humans preferred the two equally. For a method that removes human labelling from the loop, "as good as, sometimes better" is a strong result.

The result that rules out the easy explanation is the same-size one. You might think RLAIF only works because a big, capable labeller is distilling its knowledge into a smaller policy. So Lee et al. ran it with the labeller the same size as the policy: it still improved over the baseline (68% preferred over the supervised model). They pushed further to a case where the labeller and the initial policy were the exact same checkpoint (a model judging responses from itself) and still saw gains. That rules out distillation from a stronger teacher: the model is bootstrapping on its own judgement, which sets up the next idea.

Self-rewarding: the model as its own teacher

If a model can judge its own outputs, why not let it generate its own training signal and iterate?

Yuan et al. (2024) take the bootstrapping to its conclusion. A single model plays two roles: it generates candidate responses, and it scores them by prompting itself as an LLM-as-a-judge. The highest- and lowest-scored responses form a preference pair, the model is trained on those pairs with DPO (the closed-form preference objective from 6.3), and the whole loop repeats, each round producing a model that is both a better responder and a better judge. The motivation is stated bluntly in the paper's first line: to reach superhuman agents, "future models require superhuman feedback", and a frozen, human-trained reward model can never provide it because it is capped at human performance. Across three iterations a fine-tuned Llama-2-70B climbed the AlpacaEval 2.0 leaderboard past Claude 2, Gemini Pro, and GPT-4 0613.

The failure modes of AI feedback

Removing the human removes a check. Four failure modes follow.

Escape, or relocate?

A human-bounded reward model was capped at human judgement. An AI-bounded one is capped at the feedback model's judgement: better than a non-expert in some places, worse in others, and capable of being wrong in correlated, systematic ways that a diverse crowd of humans would not be. Swapping the labeller does not, on its own, let you supervise a task neither the human nor the model can evaluate; it moves the ceiling from "what a human rater can tell" to "what the feedback model can tell", which is a different ceiling, not the absence of one. Whether AI feedback can ever raise the ceiling (supervise a model more capable than its supervisor) is an open empirical question that depends on the capability in play, and it is the subject of the scalable-oversight menu (debate, amplification, recursive reward modelling, weak-to-strong) in Session 8.1. Self-rewarding is one bet that the loop can climb; the failure modes above are the reasons to hold that bet at a calibrated credence.

Questions to bring to class

• An AI judge agrees with humans about as often as humans agree with each other (~78% vs ~73–77%). Does that make it a good judge, or only an unfalsifiable one on these tasks?
• The same-size and same-checkpoint results rule out "it's just distillation". What do they show, and what would model collapse predict if you iterated them indefinitely?
• Wataoka et al. find self-preference is driven by perplexity, not self-recognition. Why does that mechanism make the bias harder to remove than if the model were recognising its own text?
• Give a concrete task where AI feedback plausibly raises the oversight ceiling, and one where it cannot. What distinguishes them?
• Self-rewarding closes the generate–judge loop on purpose. Name one external source of fresh signal you would insist on keeping in the loop, and why.

Readings