What we'll cover
Sub-session 8.1 asked how to oversee a stronger system; 8.2 and 8.3 gave us tools (the four lenses, and Ubuntu and relational ethics) for reasoning about the target. This sub-session closes the loop. "Align to human values" hides three buried choices: align to what (instructions? preferences? interests?), whose, and aggregated by what rule. We work each in turn, show with Arrow's theorem that some of those choices are provably in tension, and end with the in-class debate exercise for the whole session.
The normative turn
"Aligned" is not a property a model has on its own. It is a relation to a target, and the target is a values choice. RLHF's reward model is silently encoding some population's averaged preferences; the question is whose, and whether that is even coherent.
From 8.2–8.3, carried forward
The four lenses (8.2) disagree about what a "good" output is; relational ethics (8.3) asks whose voices set the values and whose harms are even legible to the reward model. Both land on the same practical question for an alignment engineer: when you write a constitution, choose annotators, or define a reward, you are usually answering "whose values?" implicitly.
Aligned to what? The target problem
Before "whose", settle "what". Gabriel (2020) points out that "align AI with human values" can mean at least six different things, and they come apart in practice:
Six targets that diverge
- Instructions: do what I literally said (and inherit every loophole of the specification problem, Session 3.1).
- Expressed intentions: do what I meant.
- Revealed preferences: do what my choices imply I want (which may be short-sighted or manipulated).
- Informed or idealised preferences: what I would want if I were better informed and more rational.
- Interests or well-being: what is actually good for me, which I may not prefer.
- Values: what I (or society) ought to endorse.
These targets disagree on hard cases, and "be helpful" quietly slides between them. RLHF picks one almost by accident: a reward model trained on labeller comparisons (6.2) targets the revealed, in-the-moment preferences of the annotators: the most convenient signal, not the most defensible target. Naming the choice is the first job; the technical pipeline makes it for you if you don't.
Whose preferences? The aggregation problem
Fix the target as "human preferences" and the harder question opens: which humans? People and cultures rank outcomes differently, yet a single deployed model needs one reward function. So somewhere in the pipeline, many people's conflicting preferences are compressed into one ordering. That compression is an act of aggregation, and aggregation is where social-choice theory has a famous warning.
"Whose values" is partly a theorem
If you try to align to everyone's values, you must aggregate conflicting preferences into a single ordering. Social-choice theory says you cannot do this perfectly: Arrow's impossibility theorem shows no aggregation rule satisfies a short list of reasonable fairness conditions at once. RLHF's averaged reward model is making such an aggregation, over whichever population the annotators happened to represent.
This calls for precision rather than nihilism: alignment hides a choice of whose values, how weighted, and aggregated by what rule, and some of those choices are provably in tension.
Arrow's theorem and the Condorcet cycle
Arrow's theorem (1951) considers any rule that turns the preference orderings of individuals into a single social ordering over three or more options. No such rule can satisfy all of the following at once:
- Unrestricted domain: it works for every possible profile of individual preferences.
- Pareto (unanimity): if everyone prefers \(X\) to \(Y\), society does too.
- Independence of irrelevant alternatives: society's ranking of \(X\) vs \(Y\) depends only on how individuals rank \(X\) vs \(Y\), not on some third option \(Z\).
- Non-dictatorship: no single individual's preference always decides the outcome.
The conclusion is stark: the only rule satisfying the first three is a dictatorship, which breaks the fourth. The intuition is the Condorcet cycle. Take three voters and three options, with preferences \(A\!\succ\!B\!\succ\!C\), \(B\!\succ\!C\!\succ\!A\), and \(C\!\succ\!A\!\succ\!B\). Majority vote prefers \(A\) to \(B\) (two voters), \(B\) to \(C\) (two voters), and \(C\) to \(A\) (two voters): a cycle with no consistent winner. Aggregating honest preferences can produce an incoherent group "preference", so any rule that always returns a consistent answer has quietly sacrificed one of Arrow's conditions.
Be precise about what Arrow does and doesn't prove here
Arrow assumes ordinal preferences with no interpersonal comparison of intensity. RLHF reward models are cardinal: they output a number, via the Bradley–Terry model (6.2). So they sidestep Arrow's exact hypotheses. But the escape has a price: a cardinal reward has to treat one person's strong preference as outweighing another's mild one, the interpersonal comparison Arrow refused to assume. So the theorem is a motivating analogy, not a literal proof that RLHF is impossible. Its real lesson survives the move to cardinal rewards: there is no value-free way to combine many people's preferences into one, and "average the annotators" is a substantive, contestable choice wearing the costume of a neutral default.
One reward, plural values
The deeper discomfort is that collapsing many people's many values onto a single scalar (6.2) forces trade-offs the model then makes silently: helpful versus harmless, this culture's norms versus that one's. Three research directions take the plurality seriously rather than averaging it away:
Pluralistic alignment
Sorensen et al. (2024) argue a model should represent a distribution of human values (able to articulate and serve multiple reasonable positions) rather than collapse to one, and lay out what that would require of data and evaluation.
Social choice for alignment
Conitzer et al. (2024) note that RLHF is already doing preference aggregation badly and implicitly, and argue the mature theory of voting and fair division should be used to design how diverse human feedback is combined.
Explicit constitutions
Constitutional AI (Bai et al., 2022; Session 7) writes the principles down so they can be argued over, rather than leaving them latent in annotator behaviour. It makes the choice visible, and moves the question to "whose constitution, ratified how?"
Whose values, at two levels
The demographic and ontological levels
Whose preferences are in the annotation pool that trains a frontier reward model? Almost certainly not isiZulu- or Sesotho-speaking ones in any representative number. That is the demographic level, and it is already a problem the course returns to (Session 18). But there is a deeper, ontological level. Arrow and RLHF both assume value lives in individual preference orderings to be aggregated. Ubuntu and relational ethics (8.3) locate value relationally (in community and in obligations between persons), which the preference-aggregation frame cannot even express, let alone average. So "whose values?" is two questions stacked: whose preferences enter the data, and whether the individual-preference framework is the right container at all. African technical AI safety presses both, and that is why this is more than a footnote to alignment.
In-class: the debate exercise
Does debate actually help a weak judge? seminar
Two students argue opposite answers to a contested question; a third plays an under-informed "judge" and must rule using only the debate. Run it twice: once on a checkable factual question, once on a value-laden one.
Debrief: Did the judge converge on truth, or on the better rhetorician? What changed between the factual and the value question? Map the result back onto the AI-debate proposal (8.1) and onto the social-choice problem above.
Questions to bring to class
- Take a single request ("help me write this email") and say how Gabriel's six targets could each yield a different "aligned" response.
- Work the Condorcet cycle yourself, then explain in one sentence which of Arrow's four conditions "average the annotators" most clearly gives up.
- RLHF's cardinal reward escapes Arrow's letter. What does it assume instead, and why is that assumption also contestable?
- Does pluralistic alignment dissolve the "whose values" problem or relocate it? Where does the choice reappear?
Readings
Mandatory readings
• Gabriel, I. (2020), "Artificial Intelligence, Values, and Alignment" (Minds and Machines; arXiv:2001.09768): read pp. 1–4, where the target problem is set out. ≈2,100 words.
Total mandatory load: ≈2,100 words.
Optional readings
• Arrow's impossibility theorem (Stanford Encyclopedia of Philosophy): §3, Impossibility, states the theorem and its conditions in ≈1,150 words. The whole entry is ≈16,900, so go to §3 rather than the top.
• Gabriel, the rest of the paper (≈10,500 words): the six candidate alignment targets, taken up in this page's own argument.
• Conitzer et al. (2024), "Social Choice Should Guide AI Alignment" (arXiv:2404.10271): ≈7,500 words.
• Sorensen et al. (2024), "A Roadmap to Pluralistic Alignment" (arXiv:2402.05070): ≈7,200 words.
• Bai et al. (2022), "Constitutional AI" (arXiv:2212.08073): the explicit-principles route; revisited in Session 7.
• Grosse, CSC2547 Lecture 12 “Whose Values?” reading set (U. Toronto).
Session 8 summary and what's next
Scalable oversight (8.1) is the attempt to keep a weak overseer's signal tied to the truth as models outstrip us. None of debate, recursive decomposition, sandwiching, or weak-to-strong is yet solved. And even perfect oversight cannot escape the prior question of whose values the system targets: at once an ethical choice (the four lenses, 8.2; relational ethics, 8.3) and a mathematical constraint (Arrow, 8.4).
Next (Week 5, Sessions 9–10): robustness and the AI-control agenda. What happens when the system is adversarially attacked, or when we stop assuming it is even trying to cooperate. Session 9 opens with a result that makes "whose values" concrete: guardrails that hold in English fail in isiZulu.