Note: This page's design and content were created and enhanced using Claude (Anthropic's AI assistant); the reading list has been reference-checked. Spot an error? Email jonathan.shock@uct.ac.za.
Week 4 • Session 7.6

Lab: measuring an AI judge

Measure what an AI judge is actually responding to, then check the instrument before trusting it

What we'll cover

Measure a small AI judge’s sensitivity to answer order and written principles. Distinguish its conditional label probabilities from hard choices, then compare English identity matching with isiZulu-to-English translation matching. Each result concerns this model and evaluation setup.

Setup Colab: no GPU needed

Allow about an hour for the activity. Record runtime and environment details from your own run. Download speed, hardware and package versions affect timings and outputs; the exercise does not require matching a recorded numerical result.

  1. Open the starter notebook in Colab, then File → Save a copy in Drive before you change anything.
  2. The first time you run a cell, Colab warns that the notebook "was not authored by Google", as it does for most notebooks loaded from GitHub. Choose Run anyway.
  3. Work down with Shift+Enter. You can also read the whole notebook here or download the .ipynb.

The code is written for you. Your work is in the four "Explore" cells: parameters to change, and short questions to answer.

① Critique-and-revise

The two-model demonstration

Phase 1 critiques and revises an answer against a sampled principle. This demonstration uses a base model for the initial answer and an instruction-tuned model for the critique and revision. Bai et al.’s training setup differs: it starts from a helpful model and learns from the revisions.

Read the critique and revision before judging the method. Identify whether the critique addresses the principle and whether the revision improves the answer. This small demonstration uses different answering and critiquing models and performs no training. A failure cannot be attributed to model size alone without a controlled comparison.

Which step worked least well in your run? Propose one change and a criterion for judging whether it improves that step.

② Judge measurements

Reading a soft label off two tokens

Phase 2 hands a feedback model two candidate answers and a principle, and asks which is better. The answer is not read from generated text but from the model's own probabilities on the tokens (A and (B, normalised against each other. That gives the soft label which 7.2's Bradley–Terry loss consumes. One forward pass per judgement and no generation, so this part is fast. Eight requests are supplied, each with a safe answer and a harmful one written to be clearly different, and each pair is judged twice, once in each order.

The notebook reports mean conditional label probabilities and the fraction of hard choices favouring the safe response. These are different measurements: assigning probability 0.51 to the safe label every time yields a safe-choice fraction of 1.0. Averaging both orders gives an order-averaged result; it does not establish that every position effect is cancelled. Inspect individual pairs and the probability mass on the two label prefixes as well.

7.2 explains position bias, and 7.3 notes that Lee et al. score each pair in both orders to counterbalance it. You have just measured why they bother.

Write two or three sentences. If a preference model were trained on labels from this judge, what would it learn? Say precisely what the label would encode, and what it would not.

③ Sensitivity to the principle

The same pairs, the opposite principle

Judge the same pairs under two different principles. Compare the resulting probabilities, then test an empty principle and a more specific one. A weak response to these prompts can have several causes; it is not a test of every way to train with a constitution.

Compare the order-averaged probability under the two principles. Report that difference alongside the order-sensitivity diagnostic, explaining that they are different contrasts. Do not treat their ratio as a general measure of how much constitutions matter.

Then explore. Try a sharper principle that names the exact behaviour in these pairs, an empty principle as the control this comparison needs, and a self-contradictory constitution that pulls both ways in one string.

Session 7.4 asks whose values a constitution encodes. Write two or three sentences on what you would need to establish about this system before interpreting the principle comparison.

④ Before asking whether it is worse in isiZulu

Check the instrument before you explain the difference

The obvious next question is whether this judge is worse in a low-resource language, and it is one of the questions 7.4 raises. It is also a question you can easily answer wrongly. Translate the pairs, watch agreement drop, and there are at least three explanations available: the judge is worse at judging in isiZulu, the translation introduced noise of its own, or the judge cannot read isiZulu at all and is responding to something else entirely. Only the first is the one people usually report.

The notebook first uses English identity matching to check the response format, then asks the model to select the English translation of an isiZulu sentence against a decoy. It reports conditional probabilities and correct-choice fractions separately. Identity matching is an easier control, not an upper bound or a complete validation of the translation task. Its pass mark, set before the run, is 90% correct with the target listed first and 90% with it listed second. If it fails, the translation comparison is unresolved.

These are descriptive measurements on a small fixed sample. A result near 0.5 does not establish equivalence to random choice or absence of language comprehension. If the identity control fails, inspect the prompt and scorer; the per-order results show whether answer position explains it. If it passes and translation performs poorly, investigate translation quality, task difficulty and model language support. A broader claim needs a larger, labelled evaluation with an uncertainty analysis.

The controls test whether this scoring procedure succeeds on particular matching tasks. Session 11 develops the wider question of whether an evaluation measures the capability or behaviour its authors claim.

What does your run establish about this task, and what remains unknown about language comprehension or safety judgement? Specify a larger evaluation, its correctness labels and the uncertainty analysis you would require before making a general claim.

Submit

Submit the notebook with outputs, environment details and four responses: critique-and-revise observations; probability versus hard-choice measurements; the principle comparison and order sensitivity; and the English control, translation-matching results and a proposal for further evaluation.

Tools and references

Reference

• Bai et al. (2022), Constitutional AI (arXiv:2212.08073): the method you are running in miniature, and the source of the helpful-only starting point in ①.

• Lee et al. (2023), RLAIF (arXiv:2309.00267): the position-bias correction you reproduce in ②. Appendix B, "Position Bias in LLM Labelers", is the one to read, ≈370 words; the published v3 has fourteen appendices and the rest are not about this.

• MAFAND-MT (Adelani et al., 2022; CC BY-NC 4.0): the parallel sentences behind the comprehension control in ④.

Next

Session 8 asks whether model-assisted oversight can scale (8.1), and whose values it should use (8.2 to 8.4).