Note: This page's design and content were created and enhanced using Claude (Anthropic's AI assistant); the reading list has been reference-checked. Spot an error? Email jonathan.shock@uct.ac.za.
Week 4 • Session 8.4

Whose values?

Even perfect oversight cannot escape the prior question of which values a system is aligned to

What we'll cover

An alignment target requires choices about which values to pursue, whose judgements to use and how to aggregate them. We examine these choices through Gabriel’s target distinctions, social choice and a debate exercise.

The normative turn

"Aligned" is not a property a model has on its own. It is a relation to a target, and the target is a values choice. RLHF's reward model is silently encoding some population's averaged preferences. The question is whose, and whether that is even coherent.

Sessions 8.2 and 8.3

The four lenses (8.2) disagree about what a "good" output is; relational ethics (8.3) asks whose voices set the values and whose harms are even legible to the reward model. Both land on the same practical question for an alignment engineer: when you write a constitution, choose annotators, or define a reward, you are usually answering "whose values?" implicitly.

Aligned to what? The target problem

Before "whose", settle "what". Gabriel (2020) points out that "align AI with human values" can mean at least six different things, and they come apart in practice:

Six targets that diverge

  • Instructions: do what I literally said (and inherit every loophole of the specification problem, Session 3.1).
  • Expressed intentions: do what I meant.
  • Revealed preferences: do what my choices imply I want (which may be short-sighted or manipulated).
  • Informed or idealised preferences: what I would want if I were better informed and more rational.
  • Interests or well-being: what is actually good for me, which I may not prefer.
  • Values: what I (or society) ought to endorse.

These targets disagree on hard cases, and "be helpful" quietly slides between them. RLHF picks one almost by accident: a reward model trained on labeller comparisons (6.2) targets the stated, in-the-moment preferences of the annotators (judgements they express when asked, not preferences revealed by their behaviour): the most convenient signal, not the most defensible target. Naming the choice is the first job; the technical pipeline makes it for you if you don't.

Whose preferences? The aggregation problem

Fix the target as "human preferences" and the harder question opens: which humans? People and cultures rank outcomes differently, yet the standard RLHF pipeline trains a single deployed model against one shared reward function. So somewhere in the pipeline, many people's conflicting preferences are compressed into one ordering. That compression is an act of aggregation, and aggregation is where social-choice theory has a famous warning.

"Whose values" is partly a theorem

If you align one model to everyone's values through a single shared reward, you must aggregate conflicting preferences into a single ordering. Social-choice theory says you cannot do this perfectly: Arrow's impossibility theorem shows no aggregation rule satisfies a short list of reasonable fairness conditions at once. RLHF's averaged reward model is making such an aggregation, over whichever population the annotators happened to represent.

This calls for precision rather than nihilism: alignment hides a choice of whose values, how weighted, and aggregated by what rule, and some of those choices are provably in tension.

Arrow's theorem and the Condorcet cycle

Arrow's theorem (1951) considers rules that turn complete, transitive rankings from at least two individuals into a complete, transitive social ranking over at least three options. No such rule can satisfy all of the following at once:

  • Unrestricted domain: it works for every possible profile of individual preferences.
  • Pareto (unanimity): if everyone prefers \(X\) to \(Y\), society does too.
  • Independence of irrelevant alternatives: society's ranking of \(X\) versus \(Y\) depends only on how individuals rank \(X\) versus \(Y\), not on some third option \(Z\).
  • Non-dictatorship: no single individual's preference always decides the outcome.

The conclusion is stark: the only rule satisfying the first three is a dictatorship, which breaks the fourth. The intuition is the Condorcet cycle. Take three voters and three options, with preferences \(A\!\succ\!B\!\succ\!C\), \(B\!\succ\!C\!\succ\!A\), and \(C\!\succ\!A\!\succ\!B\). Majority vote prefers \(A\) to \(B\) (two voters), \(B\) to \(C\) (two voters), and \(C\) to \(A\) (two voters): a cycle with no consistent winner. Aggregating honest preferences can produce an incoherent group "preference", so any rule that always returns a consistent answer has quietly sacrificed one of Arrow's conditions.

Be precise about what Arrow does and doesn't prove here

Arrow's theorem concerns rules that turn individual rankings into one social ranking, and RLHF's inputs are rankings of this kind: annotators' pairwise comparisons. The Bradley–Terry fit (6.2) outputs a number for each response, but that number is fitted to ordinal comparisons and used to rank responses, so it does not take RLHF outside Arrow's setting. When comparisons are elicited for uniformly random pairs of options, the unconstrained Bradley–Terry fit to pooled comparisons ranks options like the Borda count (Siththaranjan, Laidlaw & Hadfield-Menell, 2024), which violates independence of irrelevant alternatives. It also never sees strength of preference: three annotators who mildly prefer \(A\) to \(B\) outweigh two who strongly prefer \(B\). So Arrow does not show that RLHF is impossible; it shows that RLHF's aggregation, like any rule, gives up one of the conditions. There is no value-free way to combine many people's preferences into one, and "average the annotators" is a substantive, contestable choice wearing the costume of a neutral default.

One reward, plural values

The deeper discomfort is that collapsing many people's many values onto a single scalar (6.2) forces trade-offs the model then makes silently: helpful versus harmless, this culture's norms versus that one's. Three research directions take the plurality seriously rather than averaging it away:

Pluralistic alignment

Sorensen et al. (2024) argue a model should represent a distribution of human values (able to articulate and serve multiple reasonable positions) rather than collapse to one, and lay out what that would require of data and evaluation.

Social choice for alignment

Conitzer et al. (2024) note that RLHF is already doing preference aggregation badly and implicitly, and argue the mature theory of social choice, including voting, should be used to design how diverse human feedback is combined.

Explicit constitutions

Constitutional AI (Bai et al., 2022; Session 7) writes the principles down so they can be argued over, rather than leaving them latent in annotator behaviour. It makes the choice visible, and moves the question to "whose constitution, ratified how?"

Whose values, at two levels

The demographic and ontological levels

There are two questions about representation. First, whose judgements entered the annotation pool, and how well are its languages and communities represented? Establish this from dataset documentation rather than assuming the answer. Second, is preference aggregation an adequate account of the target? Individuals can express preferences about relationships and collective outcomes, but collecting those preferences may not fulfil requirements for community authority, reciprocity or shared decision-making.

In-class: the debate exercise

Does debate actually help a weak judge? seminar

Work in groups of three for 20 minutes. First use a factual claim whose answer can be checked against a short source supplied by the lecturer. The judge records an answer and confidence before hearing two 2-minute arguments, then records both again. Reveal the source and check whether the answer improved. Rotate roles and repeat on the university study-assistant scenario from 8.2, keeping the facts fixed. Record the normative premises behind the decision rather than marking one value judgement as factually correct.

Debrief: compare the factual judge’s before-and-after correctness and confidence. On the values case, assess whether the judge identified the competing premises and answered the strongest objection. Explain which observations bear on the AI-debate proposal and which would require a larger controlled study.

Questions to bring to class

Readings

Mandatory readings

• Gabriel, I. (2020), "Artificial Intelligence, Values, and Alignment" (Minds and Machines; arXiv:2001.09768): read pp. 1–4, where the target problem is set out. ≈2,100 words.

Total mandatory load: ≈2,100 words.

Optional readings

• Arrow's impossibility theorem (Stanford Encyclopedia of Philosophy): §3, Impossibility, states the theorem and its conditions in ≈1,150 words. The whole entry is ≈16,900, so go to §3 rather than the top.

• Gabriel, the rest of the paper (≈10,500 words): the six candidate alignment targets, taken up in this page's own argument.

• Conitzer et al. (2024), "Social Choice Should Guide AI Alignment in Dealing with Diverse Human Feedback" (arXiv:2404.10271): ≈7,500 words.

• Sorensen et al. (2024), "A Roadmap to Pluralistic Alignment" (arXiv:2402.05070): ≈7,200 words.

• Bai et al. (2022), "Constitutional AI" (arXiv:2212.08073): the explicit-principles route; covered in Session 7.

• Siththaranjan, Laidlaw & Hadfield-Menell (2024), "Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHF" (arXiv:2312.08358): Section 3 (Theorem 3.1) and Appendix B.2 on Bradley–Terry preference learning and Borda count.

• Grosse, CSC2547 Lecture 12 “Whose Values?” reading set (U. Toronto).

Session 8 summary and what's next

Scalable oversight (8.1) is the attempt to keep a weak overseer's signal tied to the truth as models outstrip us. None of debate, recursive decomposition, sandwiching, or weak-to-strong is yet solved. And even perfect oversight cannot escape the prior question of whose values the system targets: at once an ethical choice (the four lenses, 8.2; relational ethics, 8.3) and a mathematical constraint (Arrow, 8.4).

Next (Week 5, Session 9): robustness. What happens when the system is adversarially attacked. Session 9 starts with adversarial examples in image classifiers. Sub-session 9.3 then makes "whose values" concrete: in a 2023 study, GPT-4's safety training held for English prompts but often failed when the same prompts were translated into isiZulu.