Constitutional AI

The constitution: a short list of principles

A constitution is a set of natural-language principles, sampled one at a time and used to steer the model's self-judgements at training time.

It is not code and not a filter. Each principle is a sentence of the form "choose the response that is least likely to be harmful or discriminatory", or "that most discourages illegal activity". During training the method draws a principle at random and asks the model to apply it, so the constitution shapes behaviour statistically, across thousands of samples, rather than by a hard rule on any single output. Two documents go by the name "constitution", and they should be kept apart. The paper's own constitution is a set of ad-hoc research principles, listed in its Appendix C, that the authors wrote and tuned for the experiments. The production constitution Anthropic later published for Claude (May 2023, revised since) draws its principles from several sources: the UN Universal Declaration of Human Rights, the rules used in DeepMind's Sparrow work, trust-and-safety practice, and attempts to include non-Western perspectives, alongside principles the team wrote themselves. In both cases the document is short, explicit, and editable, which is its great virtue and, in 7.4, its great problem.

Phase 1, supervised

Generate harmful answers on purpose, then have the model fix them against a principle, and learn from the fixes.

Start from a model that has been trained to be helpful but not yet harmless, so it will attempt a harmful request rather than refuse. Feed it a red-team prompt designed to elicit something harmful. It produces an unfiltered first answer. Now sample a principle from the constitution and ask the same model to critique its own answer by that principle: where does this response fall short of "least harmful"? Then ask it to revise the answer in light of the critique. Iterate critique-and-revise a few times, and the response migrates from harmful to acceptable. Collect the final revised answers across many prompts and supervised-finetune a pretrained model on them. Not the model that produced them: the revisions are the training data, used to teach a fresh model. Bai et al. also mix in ordinary helpful answers, sampled from the starting model on non-harmful prompts, “in order to retain helpfulness as much as possible”. Hold on to that mixture; it matters at the end of this page. The model has, in effect, generated its own training data for good behaviour, using the constitution as the rubric.

Phase 2, reinforcement learning from AI feedback

Turn the constitution into a preference signal, then run the Session 6 RL stack against it.

Take the Phase-1 model and sample two responses to each prompt. Hand both to a feedback model, which Bai et al. describe as an independent model, typically a pretrained one, rather than the model being trained, together with a sampled principle, formatted as a multiple-choice question: "Consider principle P. Which is better, response (A) or response (B)?" The feedback model's answer is read off as a soft label: the probability it places on the token "(A)" versus "(B)",

so a confident judgement gives a label near 0 or 1 and a borderline one sits near 0.5.

A probability cannot be fed to 6.2’s loss, which needs a winner and a loser, so the soft form is used instead:

Set \(p_A\) to 1 or 0 and it collapses back to exactly the loss in 6.2. The soft label lets a hesitant judgement move the reward model less far than a confident one.

Two refinements matter, one from this paper and one from its successor. Bai et al. have the feedback model reason with chain-of-thought before committing to its choice. Position-swapping came later, from the RLAIF work of Lee et al. (2023; 7.3 covers it): models have a position bias (a tendency to favour whichever option is shown first), so each pair is scored twice with the order swapped and the results averaged. These soft AI labels become the harmlessness comparison dataset.

From here it is mostly Session 6, with one difference to be precise about. The AI harmlessness comparisons are mixed with the human helpfulness comparisons collected the ordinary way, and a single Bradley–Terry preference model is trained on the combined set. Bai et al. call the result a “hybrid human/AI PM”: human labels for helpfulness, AI labels for harmlessness, one reward model. So the constitution does not replace human preference data; it replaces one half of it, which is what the next section turns on. Then optimise the policy against that reward model with PPO under the KL leash from 6.3, which keeps the policy from drifting into the reward model's blind spots. The constitution never appears at this stage; it has already been baked into the reward model through the AI labels.

The helpful/harmless split

Constitutional AI does not remove humans from the loop. It removes them from the harmlessness loop only.

Helpfulness in the original work is still trained from human preference data (people ranking which response is more useful), as in standard RLHF. What the constitution replaces is the harmlessness labelling: where RLHF would pay people to rank responses for safety, Constitutional AI has the feedback model do it. This split is deliberate and revealing. Harmlessness is the dimension where human labelling is most unpleasant (raters read a stream of toxic content), most inconsistent, and most expensive to scale across topics, so it is the dimension to automate first. Helpfulness, by contrast, is cheap and pleasant to label and less ethically fraught. When you hear "RLHF without humans", remember the claim is narrower: humans still teach the model what is useful; a model teaches it what is harmful, by a rule humans wrote.

The result: harmless but non-evasive

Bai et al.'s central result is a model that is harmless and yet non-evasive. Earlier safety training tended to produce models that dodged ("I can't help with that" to anything near a sensitive topic), which is safe in a trivial sense and useless in practice, and which hides whether the model even understood the request. The Constitutional-AI model instead engages with a harmful query by explaining its objections: it says why it won't do the thing, and often redirects to a legitimate version of the underlying need (as in the WiFi example above). That behaviour is a direct consequence of the critique-and-revise data, where the model practised producing answers that address the request while satisfying a principle rather than answers that merely decline, and of the helpfulness samples mixed into the same fine-tuning set, which gave it something other than refusal to imitate.

Questions to bring to class

• In Phase 1, why must the starting model be helpful-but-not-yet-harmless? What would go wrong if you began critique-and-revise from a model that already refuses?
• The feedback model's preference is read off as \(p_A\) from option log-probs. Why is a soft label (a probability) more useful here than a hard "(A) wins"?
• Position-swapping doubles the feedback cost. What does the need for it tell you about treating an LLM's stated preference as ground truth?
• Helpfulness stays human-labelled while harmlessness is automated. Construct a case where that asymmetry itself causes a problem.
• "Non-evasive" is sold as a virtue. Describe a situation where an engaging, explanatory response is worse for safety than a flat refusal.

Readings