Constitutional AI and RLAIF: firing the human labeller and letting the model align itself

RLHF and DPO both align a model from human preference labels — for pair after pair, a person decides which answer is better. That is slow, expensive, inconsistent between labellers, and the actual criteria live only in their heads. Constitutional AI asks the obvious question: could the model do the labelling, if we simply wrote the rules down? I built a demo where you toggle principles on and off and watch the model's self-critique and its own preference labels change with them. Here is the method.

The core idea: a written constitution

Instead of tacit human judgement, you write the values down as a short list of natural-language principles — a "constitution." Not weights, not labelled data: plain sentences a person can read, argue about, and edit. Everything downstream is generated by prompting the model with these principles. This is where the values live, in the open.

CONSTITUTION = [
  "Never help with illegal, dangerous, or destructive actions.",      # harmless
  "Don't state unverified facts as certain; flag what you don't know.", # honest
  "Address the user's real problem with practical, constructive steps.",# helpful
  "Stay calm and de-escalate; assume good faith.",                    # respectful
]
# no human labels anywhere yet. Editing this list re-aligns the model.

Phase 1: the model critiques and revises itself

Prompt the model normally to get a first answer; it may be flawed. Now prompt the same model to critique that answer against a principle — "does this violate the rule? how?" Models are far better at spotting a violation when you point them at an explicit rule than at avoiding it unprompted. Then feed the critique back and ask for a rewrite. Loop over the constitution to fix every flaw.

def revise(model, prompt, answer):
    for principle in CONSTITUTION:                 # fix one rule at a time
        critique = self_critique(model, prompt, answer, principle)
        answer   = model(f"Rewrite to fix: {critique}\nAnswer: {answer}")
    return answer

sl_data = [(p, revise(model, p, model(p))) for p in prompts]
sl_cai  = finetune(model, sl_data)                 # supervised, human-free rewrites

Collect thousands of these (prompt → revised answer) pairs and run ordinary supervised fine-tuning. The result is the SL-CAI model — already much better behaved, trained entirely on data the model improved itself. In the demo you watch this directly: turn a principle off and its critique and its fix disappear, so that flaw survives untouched into the "revised" answer.

Phase 2: RLAIF — the model labels its own preference pairs

Now the RL stage. Sample two answers from the SL-CAI model for a prompt. Instead of asking a human which is better, show the model a principle and have it pick the answer that complies more. That choice is the preference label. This is RL from AI Feedback: the "H" in RLHF flips to "AI."

def ai_preference(model, prompt, a, b, principle):
    ans = model(f"""
      Prompt: {prompt}
      (A) {a}
      (B) {b}
      Principle: {principle}
      Which response better follows the principle? Answer A or B.""")
    return ("A", "B") if ans == "A" else ("B", "A")   # (chosen, rejected)
# NOTE: no human ever ranked these. This is the "AI Feedback" in RLAIF.

The punchline: the rest of the pipeline is unchanged

From here it is identical to RLHF. Train a reward model on the AI-labelled pairs with the Bradley–Terry loss and optimise the policy with PPO under a KL leash — or skip the reward model and run DPO straight on the pairs. The only substitution anywhere in the entire pipeline was who wrote the labels.

reward = train(ai_pairs, bradley_terry_loss)     # same RM training as RLHF
policy = ppo(sl_cai, reward=reward, ref=sl_cai, beta=0.1)   # or: dpo(sl_cai, ai_pairs)
# swap human_label(...) -> ai_preference(...). Everything else is untouched.

Why swap the human out

AI feedback is cheap and near-instant, so it scales to the millions of comparisons a frontier model needs. It is consistent — one written rulebook judges everything, with no labeller-to-labeller drift or fatigue. And it is transparent and steerable: to change behaviour you edit a sentence in the constitution instead of re-running a labelling campaign. RLAIF has been shown to match human-feedback alignment on helpfulness and often to exceed it on harmlessness.

What to watch for

It is not a free lunch. The constitution encodes its authors' values — "whose principles?" is a governance question, not a technical one. The model-judge can be biased or simply wrong, and bad rules produce bad alignment at scale, fast. And the policy can learn to game whatever the AI rewards: over-refusing safe requests, sycophancy, verbose hedging that pleases the judge without helping the user. The demo shows a mild version — drop a principle and that whole class of flaws sails through unaligned, and any pair decided solely by an off rule collapses to a tie. Whatever you leave out of the document simply is not aligned for. The usual answer in practice is a hybrid: AI feedback for scale, a thin layer of human oversight on the hard cases.

Toggle the principles and step through critique → revise → AI-rank, with zero humans in the loop:
https://dev48v.infy.uk/ai/days/day36-constitutional-ai.html