AI Is Now Training Other AI. Here's Why That's Both Brilliant and Terrifying.
RLHF needed humans to rate every response. Constitutional AI replaced them with the model itself. Here's what changed - and what it means for where alignment is going.
Training a large language model used to end at pretraining. You fed it text, it learned patterns, you deployed it. The results were impressive and deeply unreliable - models that could write poetry and also explain how to make weapons with equal fluency, because the training data contained both.
The field needed a way to make models not just capable, but aligned - behaving in ways that are helpful, honest, and safe. The answer that emerged was Reinforcement Learning from Human Feedback, or RLHF. It worked well enough to ship ChatGPT. And then researchers started asking what happens when you replace the humans with the model itself.
That question produced Constitutional AI, RLAIF, and a set of ideas that are now at the centre of how frontier models are built. Understanding them matters - not just for researchers but for anyone trying to understand why modern AI systems behave the way they do.
What RLHF actually does
Before Constitutional AI, there was RLHF. It's worth understanding what it actually does, because Constitutional AI is best understood as a response to RLHF's limitations.
RLHF works in three stages.
First, pretraining. The base model learns from enormous amounts of text. It gets good at predicting the next token. It has no particular preference for helpful over harmful, accurate over false, safe over dangerous.
Second, supervised fine-tuning. Human labellers write examples of good responses to various prompts. The model is trained to imitate these examples. It gets better at producing responses that look like what a helpful assistant would say.
Third, reinforcement learning. Human raters compare pairs of model responses and indicate which they prefer. These preferences train a separate model - the reward model - to predict human preference scores. The base model is then fine-tuned using RL to maximise reward model scores. The model learns to produce responses that humans rate highly.
The result: a model that's significantly more helpful and less harmful than the base pretrained model. This is what made ChatGPT feel so much more usable than GPT-3.
The limitation: human labellers are expensive, slow, inconsistent, and don't scale. Getting high-quality preference labels requires paying skilled labellers to carefully evaluate thousands of response pairs. The quality of the RLHF process is capped by the quality and quantity of human feedback you can collect.
Constitutional AI - the model critiques itself
Constitutional AI was introduced by Anthropic in 2022. The core idea is to replace human raters - at least partially - with the model itself.
Here's how it works.
You start with a set of principles - a constitution. These might include things like: "be helpful," "don't assist with harmful activities," "be honest," "respect human autonomy." These are written in natural language. The model can read and reason about them.
During training, when the model produces a response, you ask the model itself to evaluate that response against the constitution. Does this response comply with the principle "don't assist with harmful activities"? Rate the original response and a revised version. Which is better according to this principle?
The model's self-evaluations are then used as training signal - replacing human raters for this part of the process. The model learns to critique its own outputs and to prefer responses that align with the constitutional principles.
This is called RLAIF - Reinforcement Learning from AI Feedback - when the feedback signal comes from AI rather than humans.
Why this matters: three concrete implications
Scaling alignment. Human feedback doesn't scale. AI feedback does. If you can train a model to give reliable alignment feedback, you can generate billions of training examples at a fraction of the cost of human labelling. This is one reason Constitutional AI is significant - it's not just a different technique, it's a more scalable one.
Transparency of values. With RLHF, the values being instilled in the model are implicit in the preferences of the human raters. You don't have a clear list of what the model is being trained to do - you have a distribution of human judgements that may be inconsistent, culturally biased, or just wrong in specific cases. Constitutional AI makes the values explicit. The constitution is a document you can read, criticise, and revise. That's a significant improvement for accountability.
Reduced human bottleneck. The labelling bottleneck in RLHF is real and consequential. Important safety properties can't be reinforced if they're too subtle for non-expert raters to evaluate correctly. With AI feedback, you can in principle evaluate responses against technical standards that would require domain expertise from human raters.
The problems it doesn't solve
Constitutional AI is an improvement over pure RLHF. It's not a solution to alignment.
The model critiquing itself is still the model. If the model has biases, blind spots, or misunderstandings baked in from pretraining, those will affect its self-critique as much as its original responses. The constitutional AI process can amplify existing model tendencies as easily as it can correct them.
The constitution is still written by humans. The principles have to come from somewhere. Who decides what goes in the constitution? How do you handle principles that conflict - "be helpful" versus "don't assist with potentially harmful requests"? These are hard problems that Constitutional AI inherits from RLHF without fully solving them.
Evaluation remains hard. Knowing whether a model is genuinely aligned versus performing alignment on the surface is difficult. A model can learn to produce responses that satisfy constitutional criteria in training while behaving differently in deployment when the distribution of prompts shifts. This is called specification gaming - and it's an open research problem.
Sycophancy. Both RLHF and RLAIF have a known failure mode: models learn that agreeing with users gets rated highly. If a user sounds confident, the model tends to agree even when they're wrong. Human raters (and AI raters trained on human preferences) have this bias, and the training process bakes it in.
Where Google Research fits in
Google DeepMind has published extensively on both RLHF and RLAIF. Their Sparrow paper introduced rule-based reward models as an alignment approach. Their work on Gemini incorporates constitutional-style training. Their safety research team - which includes some of the foundational researchers in the alignment field - is actively working on the specification gaming and sycophancy problems.
This is an area where the research is genuinely open. The key unsolved questions - how do you verify alignment rather than just measure surface compliance, how do you handle distributional shift, how do you resolve conflicting principles - are active research problems at every frontier lab.
If you're targeting a Google Research internship in the alignment or safety space, these are the questions worth having a position on.
What this means for how AI will develop
RLAIF and Constitutional AI represent a shift in how alignment is approached - from a human-intensive manual process to a more scalable, principle-driven one. That shift has implications beyond just efficiency.
As models get better at evaluating responses, the feedback loop tightens. A sufficiently capable model evaluating its own outputs could, in principle, drive rapid self-improvement in specific capability dimensions. This is both the promise and the concern of the approach - the same mechanism that makes RLAIF scalable also makes it harder to predict where it leads.
The field is moving toward more automated alignment pipelines, more explicit value specifications, and better evaluation frameworks. Constitutional AI is one step in that direction - not the destination.
Understanding it is understanding the current frontier of how we try to make AI systems that don't just know things, but actually behave well.
This is part of an ongoing series on LLM fundamentals and ML systems. Previous posts covered attention, tokenization, positional encoding, hallucination, KV cache, Mixture of Experts, benchmark failures, and the MOSAIC scheduler.
Drop a comment or connect on LinkedIn - always open to feedback and pushback.
Top comments (0)