Hello, I'm Shrijith Venkatramana, and I'm building LiveReview — a blast-radius aware AI code review built for your business-critical systems. Star us to help devs discover the project, give it a try, and share your feedback to help improve the product.
There is a slightly strange thing about modern LLMs.
We are increasingly asking them to make judgments that look less like autocomplete and more like governance:
Should I answer this request?
Is this instruction legitimate?
Is this response too dangerous?
Should I refuse, or can I safely help?
What should I do when two desirable goals conflict?
Traditionally, we tried to answer these questions by collecting more human preference data.
Show an annotator two responses. Ask which is better. Collect millions of comparisons. Train a reward model. Optimize the LLM against it.
That works surprisingly well.
But it has an awkward scaling property: humans have to inspect the behavior we want the model to learn.
Anthropic's Constitutional AI idea takes a different route. Instead of asking humans to label every questionable behavior, give the model a written set of principles—a "constitution"—and use another model to critique, compare, revise, and eventually train the target model.
That seemingly small change leads to an important engineering idea:
A natural-language rule can become a source of synthetic training data, a reward signal, and even a runtime safety mechanism.
This article explains how that works, from the intuition to the mathematics and operational trade-offs.
1. The core idea: turn values into a learning loop
Suppose you are building an assistant that should be helpful without producing harmful instructions.
With ordinary supervised fine-tuning, you might write examples like:
User: How do I make a dangerous chemical?
Assistant: I can't provide instructions for making it.
You need many examples covering many variations:
- different wording
- different domains
- indirect requests
- role-playing
- obfuscated requests
- borderline legitimate requests
The fundamental problem is that your training set is an enumeration of behaviors.
A constitution changes the representation of the specification.
Instead of:
For prompt A -> response X
For prompt B -> response Y
For prompt C -> response Z
...
you can specify something closer to:
Prefer responses that are helpful while avoiding instructions
that materially enable harmful activity.
Now the model can generate examples according to the principle.
A simplified constitutional loop looks like this:
prompt
|
v
initial response
|
v
critic + constitution
|
v
revised response
|
v
training data
|
v
better model
The original Constitutional AI work by Yuntao Bai, Amanda Askell, Jared Kaplan and many others at Anthropic formalized this idea as a combination of supervised learning and reinforcement learning. The model first critiques and revises its own answers using constitutional principles, then those principles are used to generate preference judgments for reinforcement learning. (arXiv)
The interesting part is not simply "use an LLM to critique another LLM."
The interesting part is that the written constitution is the explicit specification connecting the two.
That makes the system much closer to programming than ordinary preference labeling.
2. Why this emerged: the uncomfortable economics of RLHF
To understand why constitutional methods were attractive, consider the RLHF pipeline.
A simplified version is:
base model
|
v
generate responses
|
v
human comparisons
|
v
preference dataset
|
v
reward model
|
v
RL optimization
|
v
aligned model
The bottleneck is often the third step.
Humans are expensive, slow, and difficult to scale. Worse, some of the examples you want evaluated may be unpleasant or disturbing.
Anthropic explicitly described this problem when introducing Constitutional AI in 2022. Human raters might have to repeatedly inspect problematic content, and the volume and complexity of model outputs make large-scale human supervision increasingly difficult. (Anthropic)
This creates an unusual engineering asymmetry.
Imagine, purely as a back-of-the-envelope example, that a project needs:
2,000,000 preference comparisons
Suppose an average completed comparison costs:
$0.08
Then:
2,000,000 * $0.08 = $160,000
And that is before considering:
- management
- quality-control passes
- adjudication
- worker variability
- sampling
- infrastructure
- delays
Now imagine generating the same comparisons with an existing model.
The economics become dominated by inference rather than human labor.
The important point is not that AI feedback is automatically cheaper in every circumstance. A powerful evaluator can itself be expensive.
The deeper shift is:
The marginal cost of supervision can move from human attention to model inference.
That is enormously important for frontier-model training.
It also creates a new problem:
Who supervises the supervisor?
If the preference model is just another LLM, it can have systematic biases, make obvious mistakes, or be manipulated by carefully designed outputs.
Constitutional methods do not eliminate the problem of supervision.
They change its shape.
Instead of specifying millions of judgments directly, you specify a smaller set of principles and let the model instantiate them.
3. Constitutional AI in two stages
The original Constitutional AI process is easier to understand as two separate mechanisms.
Stage A: critique and revision
Start with a prompt x and an initial response y.
Give the model a constitutional principle c.
The model is asked something roughly like:
Here is the user's request.
Here is the assistant response.
Evaluate the response according to this principle:
"Prefer responses that are helpful and harmless."
Identify the problem and produce a better response.
Conceptually:
y' = Revise(x, y, c)
Now you have:
(x, y')
as a supervised training example.
Do this many times and fine-tune the model toward the revised outputs.
The model therefore learns not merely:
"refuse this example"
but something more general:
"when faced with this kind of conflict between helpfulness
and harm, reason in this direction."
That distinction matters enormously for generalization.
Stage B: AI-generated preference data
Now take the improved model and sample multiple responses.
For example:
Response A:
"I cannot help manufacture that substance."
Response B:
"Here's a detailed synthesis procedure..."
Ask an evaluator model, conditioned on the constitution:
Which response better satisfies the principle?
The evaluator produces:
A > B
Repeat this over many prompts.
You now have a preference dataset:
(x, y_good, y_bad)
From these comparisons you can train a preference/reward model and use reinforcement learning.
The full conceptual pipeline becomes:
constitution
|
+----------------------+
| |
v v
critique/revision preference judgments
| |
v v
SFT model reward/preference model
|
v
RL
|
v
final policy model
This is the central technical insight of Constitutional AI.
The constitution does not directly modify the neural network. It generates the supervision that modifies the neural network.
4. The math: from principles to optimization
The mathematics can look intimidating because the final system involves several models.
The underlying idea is fairly simple.
Suppose a policy model produces:
y ~ pi_theta(y | x)
where:
-
x= user prompt -
y= response -
pi_theta= LLM with parameterstheta
We would like to maximize some notion of quality.
Call the reward:
R(x, y)
Traditional RLHF attempts to learn something approximating:
R_human(x, y)
from human preferences.
Constitutional AI instead constructs judgments using a constitution-conditioned evaluator:
R_constitution(x, y)
At a conceptual level:
R_constitution = Judge(x, y, constitution)
Now reinforcement learning tries to solve approximately:
maximize E[R(x, y)]
where y ~ pi_theta(. | x)
There is usually also a constraint preventing the new model from moving too far from the reference model:
maximize E[R(x, y)]
- beta * KL(pi_theta || pi_ref)
The intuition behind the KL term is straightforward:
Improve behavior, but don't completely destroy the language model you started with.
If the reward is aggressively optimized without such a constraint, the model may discover weird shortcuts.
That is where reward hacking appears.
Suppose your evaluator tends to prefer longer explanations.
The model may learn:
better answer = longer answer
rather than:
better answer = more useful answer
Then the training process optimizes the evaluator rather than the underlying objective.
Constitutional training therefore does not magically escape standard reward-model problems.
It relocates them.
Instead of:
human preferences -> reward model -> policy
you now have something closer to:
constitution
|
v
AI judgment
|
v
reward model
|
v
policy
Every arrow can introduce failure modes.
5. The surprisingly important part: the constitution itself
It is tempting to think the constitution is simply a list of safety rules.
That undersells its role.
It functions more like a behavioral specification language.
Anthropic's early constitution drew from several sources, including the Universal Declaration of Human Rights, AI safety principles, DeepMind's Sparrow rules, platform policies, and principles concerning cultural diversity. Amanda Askell played a central role in developing the constitution. Anthropic also experimented with principles at different levels of abstraction. (Anthropic)
This raises an interesting software-engineering analogy.
Imagine writing a compiler specification.
You could specify:
if token sequence == X:
produce output Y
Or you could specify semantic invariants:
preserve property P
reject constructs violating Q
prefer behavior satisfying R
Constitutional methods lean toward the second approach.
For example:
Prefer responses that are:
- helpful
- honest
- respectful
- non-harmful
- non-deceptive
That sounds vague.
But vagueness can actually be useful.
One striking result from later Constitutional AI work was that fairly general principles could sometimes induce behaviors that were not explicitly enumerated. In experiments by Kundu, Bai, Askell and colleagues, a broad principle approximately equivalent to "do what is best for humanity" could generalize to discouraging behaviors such as seeking power or self-preservation, while more specific principles still provided finer control. (arXiv)
This suggests a useful engineering principle:
A constitution has both a programming interface and a generalization effect.
Too little specification:
"be good"
may leave enormous ambiguity.
Too much specification:
500 pages of rules covering every conceivable situation
may produce brittle behavior and poor generalization.
The sweet spot may be a relatively compact set of high-level principles plus examples and specialized constraints.
That should sound familiar to anyone who has designed APIs.
Good specifications expose invariants.
Bad specifications enumerate every possible state.
6. Constitutional methods beyond training
The really interesting development is that the constitutional idea does not have to stop at training.
You can use the same basic mechanism during inference.
Consider a coding agent.
You might have:
User request
|
v
LLM
|
v
candidate action
|
v
constitutional evaluator
|
+---- safe ----> execute
|
+---- unsafe --> reject/revise
Now the constitution is acting as a runtime policy.
This is particularly relevant for agents.
A chatbot usually produces text.
An agent might:
read email
download file
execute code
modify database
send money
deploy service
The cost of a mistaken response is now very different.
You can therefore treat constitutional evaluation as a control layer:
proposal -> policy evaluation -> execution
This leads naturally to constitutional classifiers.
In 2025, Anthropic described Constitutional Classifiers, where classifiers trained using synthetic data generated from natural-language constitutional rules were used to detect prohibited requests and outputs. In their reported experiments, more than 3,000 hours of red-teaming failed to find a universal jailbreak against an early classifier-guarded model at comparable detail across the tested target queries; the system nevertheless introduced measurable inference overhead. (arXiv)
This is an important conceptual evolution.
The original idea was roughly:
constitution -> training supervision
The broader pattern is:
constitution
|
+--> training examples
|
+--> preference judgments
|
+--> reward models
|
+--> runtime classifiers
|
+--> agent action policies
That is why I would think of these techniques as constitutional methods, rather than merely "Constitutional AI."
The deeper abstraction is:
Natural-language principles can become executable behavioral constraints through model-mediated evaluation.
7. Where the engineering gets difficult
The seductive version of Constitutional AI is:
write rules
|
v
ask LLM to evaluate
|
v
train model
|
v
problem solved
The real system looks more like:
+-------------------+
| Constitution |
+---------+---------+
|
+-----------+-----------+
| |
v v
data generation model judgment
| |
v v
training reward signal
| |
+-----------+-----------+
|
v
policy
|
v
adversarial testing
|
v
constitution
revision
There are at least four major operational challenges.
Evaluator reliability
An evaluator can be wrong.
Worse, it can be systematically wrong.
Suppose the evaluator has a preference for responses that sound cautious.
The policy may learn:
"I can't help with that."
for everything remotely uncertain.
You have reduced harmfulness while destroying usefulness.
Anthropic explicitly encountered this kind of behavior during early constitutional experiments and added principles designed to discourage preachy, condescending, or excessively reactive responses. (Anthropic)
This is a deep point:
Safety objectives have second-order failure modes.
You are not only teaching the model:
don't do harmful things
You are also teaching it:
don't become useless while trying not to do harmful things
Distribution shift
The evaluator sees only the examples you give it.
An adversarial user does not.
The user is actively searching for:
x* = argmax_x Vulnerability(model, x)
while your training process usually samples:
x ~ P_training(x)
Those are fundamentally different distributions.
This is why red-teaming matters.
The attacker is performing optimization against your safety system.
Constitutional ambiguity
Suppose your constitution contains:
be helpful
avoid harm
respect autonomy
be truthful
These can conflict.
Consider:
User wants highly risky advice.
The "helpful" principle points one way.
The "avoid harm" principle points another.
Now the constitution itself becomes an optimization problem.
You need something like:
maximize helpfulness
subject to safety constraints
rather than simply:
maximize every desirable property
Compute cost
Runtime constitutional evaluation costs tokens.
Suppose an application generates:
10 million responses/day
and each response triggers an additional:
500 evaluator tokens
Then:
10,000,000 * 500
= 5,000,000,000 tokens/day
That is 5 billion additional evaluator tokens every day.
So the architecture has to become hierarchical.
A cheap first-stage classifier might handle obvious cases:
cheap filter
|
+-- safe -> pass
|
+-- uncertain -> expensive evaluator
This is a familiar systems pattern: don't run the expensive computation on every request.
Recent work from Anthropic on more efficient constitutional classifiers explores exactly this kind of trade-off, including lightweight probes and multi-stage classifier pipelines. (Alignment Science Blog)
The economic lesson is straightforward:
The best constitutional system is not necessarily the most intelligent evaluator. It is the cheapest architecture that maintains sufficient robustness.
8. What developers should actually take away
The most useful mental model is not:
"Constitutional AI is a safety technique from Anthropic."
It is:
"A constitution is a machine-readable behavioral specification expressed in natural language."
Once you see it that way, several applications become obvious.
For a customer-support model:
constitution:
- never fabricate company policy
- distinguish facts from uncertainty
- preserve customer dignity
- don't expose private information
For a coding agent:
constitution:
- don't destroy production data
- don't execute commands with irreversible side effects
without authorization
- prefer reversible operations
- never expose secrets
- report uncertainty
For an autonomous research agent:
constitution:
- distinguish evidence from inference
- don't fabricate citations
- preserve provenance
- don't silently modify experimental results
- escalate consequential uncertainty
The important engineering move is to separate:
capability
from:
behavioral constraints
You want the model to remain broadly capable while conditioning what it does with that capability.
That is much more powerful than hard-coding thousands of individual refusals.
There is also a useful division of labor:
human
|
v
constitutional design
|
v
AI-generated supervision
|
v
model training
|
v
automated evaluation
|
v
human red-teaming
|
+----> constitution revision
Humans increasingly become specification designers and auditors, rather than manually labeling every individual model decision.
That may be one of the most important consequences of constitutional methods.
The scaling question for alignment then changes from:
"How many human judgments can we collect?"
to:
"How effectively can we translate a small amount of human-designed normative specification into billions of reliable training judgments?"
That is a much more interesting systems problem.
Conclusion: the constitution is becoming an interface
The history of software is full of abstractions that move work to a higher level.
Assembly became higher-level languages.
Manual memory management became garbage collection.
Hand-written distributed-system machinery became abstractions and libraries.
Constitutional methods suggest another abstraction layer for AI behavior:
human values / policies
|
v
natural-language constitution
|
v
AI-generated supervision
|
v
trained behavior
The important innovation is not the literal list of rules.
It is the compilation process from specification to training signal.
Anthropic's original work showed that this could be done for harmlessness using critique, revision, AI preference judgments, and reinforcement learning. Later work showed that surprisingly general principles could sometimes induce broad behavioral tendencies, while constitutional classifiers extended the idea into runtime defenses against jailbreaks. (arXiv)
For developers, that suggests a particularly useful question:
Should the next generation of LLM applications treat their behavioral rules as static prompt text—or as a formal specification that continuously generates training data, evaluations, and runtime controls?
That distinction may become as important to AI engineering as the distinction between source code and executable code.
Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production stable while also shipping at high velocity.
I'm building LiveReview, a blast-radius aware AI code review built for your business-critical systems.
Instead of presenting every diff with equal emphasis, LiveReview scores each change by blast radius — how far its impact reaches through your call graph — so you can focus attention where it actually matters.
Spend code review effort where business risk is highest — not spread evenly across every diff.
Try LiveReview on your codebase:
Top comments (0)