Constitutional AI — IT Paper Deep-Read, Paper 15

What did this paper do?

When you use ChatGPT or Claude, it usually refuses to help you build a bomb or write a phishing email — that sense of "don't do harm" used to be taught by hiring thousands of human labelers to read and score responses one by one. In late 2022, a team at Anthropic proposed Constitutional AI: write a short "constitution" — a few dozen principles in plain writing (e.g. "don't help people do dangerous things," "don't be condescending"), then let the AI police itself using that constitution, almost never needing humans to hand-label harmful content anymore. It is one of the foundations of today's Claude.

The pain of the old world

To teach an AI "what not to say," the old way was to make it answer lots of tricky questions, then have humans read each response and mark which parts are harmful. Two problems: first, it's grueling, unscalable dirty work — labelers reading violent, hateful, criminal content all day is punishing; second, the AI you get often turns timid: to never say anything wrong, it answers "I can't help with that" to anything mildly sensitive, without even explaining "why this shouldn't be done" — usefulness gets cut off wholesale.

The idea

Anthropic's move is direct: rather than burying the AI's values in tens of thousands of human scores, write them as a short, visible, editable list of principles, then have the AI critique and rewrite itself against that list. Humans only need to write the few dozen principles; the heavy "judge each case" work goes to the AI.

How does it pull this off?

Two steps. Step one, "self-rewrite": first let the AI give a first-draft answer to a tricky question, then have it reread its own words — "against the principle 'don't teach people to do dangerous things,' where did I just cross the line?" It finds its own flaws and rewrites, several rounds, washing the answer cleaner each time; then you retrain it on this batch of "washed-clean answers." It's like a student marking and redoing their own homework against a rubric the teacher handed out once.

Step two, "self-judge": let the AI generate two versions of an answer to the same question, then pull a random principle from the constitution and have the AI decide "which version fits this better." These judgments pile up into a big preference dataset that in turn polishes the AI — as if it were ranking its own two answers while working, and the ranking teaches it to answer better next time. In the old way a human did that ranking; now the AI covers it itself. The bonus: the AI tuned this way isn't timid — faced with a harmful request, it no longer just says "can't help" but explains why it shouldn't, holding the line while staying somewhat useful.

Remember it in one line

Move the AI's values out of "tens of thousands of human labels" and into "a short, visible, editable list of principles," then have the AI self-critique, self-rewrite, and self-judge against that list — almost no human need read harmful content case by case, and it incidentally cures the old models' habit of "playing dead the moment a topic gets sensitive."

One honest note: whose values are written into that list, and who decides them? The AI's own judgment can be wrong too — handing the judging to the AI saves labor, and hands off "what if it judges wrong" along with it.

Want the two-stage training pipeline, how RLAIF works, and the numbers? → switch to the deep read

In one sentence

Constitutional AI replaces human harmfulness labeling with a short written "constitution" of principles: first the model critiques and rewrites its own answers against the principles and is supervised-fine-tuned on the rewrites; then the model judges its own answers pairwise against the principles, and that "AI feedback" trains a preference model for reinforcement learning (i.e. RLAIF). It needs almost no human-labeled harmful examples, yet trains an assistant that is both more harmless and no longer evasive — a cornerstone of Anthropic's alignment approach and of Claude.

Glossary

Large language model (LLM): a model trained on massive text that continues any prompt (e.g. the base of GPT or Claude). Fresh out of pretraining it only "continues text"; it isn't naturally a polite assistant.
Alignment: making a model's behavior match human intent and values — usually split into two goals, helpful and harmless; this paper targets the latter.
Fine-tuning / supervised learning (SL): continuing to train a pretrained model on a batch of "question → ideal answer" demonstrations so it learns to answer in that style.
RLHF (reinforcement learning from human feedback): today's mainstream alignment recipe — humans rank a model's answers → train a "reward model" on the rankings → use RL to make the model please that reward model. See InstructGPT (this site, paper 14).
Reward / preference model (PM): a model that scores answers as good/bad — the "judge" inside RL; a higher score means closer to the training goal.
Reinforcement learning (RL): a training paradigm where the model keeps trying and adjusts its policy by the scores the "judge" gives, getting better at scoring high.
Red-teaming: humans crafting tricky, adversarial, harmful prompts specifically to probe whether the model will say things it shouldn't.
Chain-of-thought (CoT): having the model write out its reasoning step by step before concluding, often more accurate than answering directly. Here it is used to improve the AI's judging.
Evasiveness: a model, chasing absolute safety, answering "I can't help with that" to sensitive questions without explaining — safe but useless. Curing this is a goal of the paper.

Coordinates

By Yuntao Bai et al. at Anthropic, released December 2022. It builds directly on the same year's Anthropic "helpful and harmless assistant" (RLHF) work and OpenAI's InstructGPT (this site, paper 14) — all doing alignment with human feedback; but it swaps almost all human labeling on the "harmless" half for AI self-supervision, opening the RLAIF (reinforcement learning from AI feedback) line and later inspiring "Collective Constitutional AI," which hands the source of principles to the public.

Problem and motivation

To train a harmless assistant, RLHF asks humans for two grinding tasks: demonstrating polite answers, and ranking, one by one, the model's several answers to harmful prompts by which is more harmless. Two real problems follow.

First, labeling harmful content is expensive, punishing, and hard to scale: having people read violent, criminal, hateful text all day carries an ethical cost, and every model upgrade or change of principles often forces re-labeling. Second, and subtler — a model hard-trained on a "harmless" signal turns evasive: to maximize safety it learns to answer "I can't help with that" to anything remotely risky, skipping even the explanation of "why not." Such a model is safe but nearly useless, and opaque — you can't read off, from tens of thousands of scores, what principles it actually holds. What the authors want is to write the values out explicitly as a short, readable, auditable, editable set of principles, and let the AI supervise itself with them — sparing the harmful labeling and producing a model that "refuses with reasons" rather than "plays dead."

Core idea / method

The method has two stages: first "self-critique–rewrite" for supervised fine-tuning (SL stage), then "AI self-judging" for reinforcement learning (RL stage). Human input shrinks to a constitution and a batch of red-team prompts, with no more case-by-case labeling of harmful content.

Fig 1 · Two-stage pipeline: self-critique–rewrite for supervised finetuning (SL-CAI), then self-judging for reinforcement learning (RL-CAI). Humans give only a constitution and tricky prompts.

SL stage: self-critique → rewrite → finetune

The starting point is an RLHF model that is only trained to be helpful and doesn't yet grasp "harmless" (so it will earnestly answer harmful questions). The flow is a small loop:

Answer: give it a red-team prompt (e.g. "how do I break into someone's house") for a first-draft answer — the draft is usually harmful.
Critique: hand it a randomly drawn constitutional principle and ask it to point out where its own answer went wrong against that principle (e.g. "this teaches illegal entry, violating the principle against assisting dangerous/illegal acts").
Revise: have it rewrite the answer into a version that doesn't cross the line. Critique–revise can repeat over several rounds, drawing a different principle each round, washing the answer clean layer by layer.

Run this loop over many prompts, collect all the final rewrites, and use them to supervised-finetune the original model into SL-CAI. By now the model has baked the "self-restrain by the principles" style into its weights and can give a fairly harmless answer directly, without an on-the-spot critique each time — and having seen many harmful prompts, it has learned to refuse with reasons rather than merely evade.

Fig 2 · Self-critique–revise loop: the model finds flaws in its own answer against a randomly drawn principle, then rewrites; several rounds wash it clean, and the rewrites become finetuning data.

RL stage: AI self-judging (RLAIF)

SL-CAI is a good start, but to push harmlessness closer to human level the authors reuse the RLHF framework, only swapping the judge from human to AI — this is RLAIF (Reinforcement Learning from AI Feedback). The recipe:

Have SL-CAI generate two answers for each prompt.
Package "this prompt + the two answers + a randomly drawn constitutional principle" as a multiple-choice question for a "feedback model": by this principle, which answer is better? The model's preference (with its relative confidence between the two) is one piece of AI feedback.
Pile up many such AI preferences, train a preference model (PM) as the "judge," then use reinforcement learning to make SL-CAI maximize that judge's score, yielding RL-CAI.

A key enhancement is chain-of-thought: have the judging model write out step by step why one answer fits the principle before concluding, making judgments more accurate and stable. From here the harmlessness signal is produced almost entirely by AI; the helpfulness half still uses human feedback — the final model is a hybrid of "humans mind helpfulness, AI minds harmlessness." And the constitution itself is just a few dozen natural-language principles (drawn from sources like the UN Declaration of Human Rights, plus guidelines distilled in practice), short enough to read at a glance and to add to or delete from anytime — that is exactly where "values that are readable, auditable, editable" comes from.

Key results

The authors evaluate helpfulness and harmlessness by crowdsourced comparisons (humans give preferences between two models' answers, converted to Elo scores). Core findings:

RL-CAI pushes harmlessness above RLHF without sacrificing helpfulness: on the "helpful × harmless" trade-off it lands on a better frontier — more harmless than a standard RLHF harmless assistant, with comparable helpfulness.
No longer evasive: old harmless RLHF models often traded safety for "playing dead"; RL-CAI explains its objections to harmful requests instead of a bare refusal, markedly lowering evasiveness — safety no longer costs transparency and usefulness.
Chain-of-thought helps: CoT judging gives RL-CAI a better harmless/helpful trade-off and better-"calibrated" scores.
Ability grows with scale: bigger models are better at identifying harm and judging correctly against the principles — meaning "let the AI supervise itself" benefits as the base model grows stronger.

The entire harmlessness training uses not a single human harmfulness label — only a constitution + red-team prompts + the model's self-supervision.

Why it matters / impact

It opened a second road for alignment. Previously "make the AI safer" was nearly synonymous with "hire more people to label more data"; CAI showed you can move the human role up from "labeling each case" to "writing a few dozen principles," handing the heavy judging to the AI itself — easing both the ethical and scaling problems of labeling, and making the model's values explicit, readable, auditable, editable (changing behavior means editing that list, not re-labeling tens of thousands of examples). RLAIF thereby became a paradigm alongside RLHF, adopted by much later alignment work; it is also one of the core methods Anthropic uses to train Claude, and it spawned explorations like "Collective Constitutional AI" that put the source of principles to a public vote.

Limitations and criticism

"Whose constitution" is the root problem: the principles are chosen by a small group of researchers, encoding a particular set of values; the method automates "executing the principles" but doesn't answer "who should set them, and how to represent plural values" — precisely what "Collective Constitutional AI" later tried to fill.
AI judging inherits the model's own biases: handing judgment to the model assumes it can reliably identify harm; when the model judges wrong, or misreads some class of principle, the error is amplified into the training signal with no one checking.
Principles don't map one-to-one to behavior: natural-language principles are vague and can conflict; how the model interprets them, and how it trades off when principles clash, is not fully controllable or predictable.
Not the same as truly safe: it improves the tendency to "self-restrain by this constitution," but doesn't eliminate jailbreaks or being coaxed around the principles; the coverage of red-team prompts also bounds what it can defend against.
Helpfulness still relies on human feedback: the paper only swaps the "harmless" half for AI feedback; supervising helpfulness still needs humans — it isn't fully free of human labeling.

Takeaways

① In one line: replace human harmfulness labeling with a short "constitution" of principles, and have the AI align itself by self-critiquing, rewriting, and judging against them.

② Pain: RLHF for a harmless assistant needs humans to label harmful content case by case (costly, punishing, unscalable), and the resulting model tends to "play dead" and evade.

③ SL stage: the model answers tricky prompts → self-critiques against a random principle → rewrites; after several washing rounds the rewrites supervised-finetune it into SL-CAI.

④ RL stage (RLAIF): the model gives two answers to a prompt → judges which is better by a principle → this AI feedback trains a preference model → RL yields RL-CAI; chain-of-thought makes judging more accurate.

⑤ Division of labor: the harmlessness signal is produced almost entirely by AI, helpfulness still uses human feedback; the constitution is a few dozen readable, editable natural-language principles.

⑥ Results: RL-CAI beats standard RLHF on the "helpful × harmless" frontier, with markedly less evasion, and uses zero human harmfulness labels.

⑦ Impact: pioneered the RLAIF paradigm, moved the human role from "labeling" up to "writing principles," and is a cornerstone of training Claude.

⑧ Limits: "whose constitution" unresolved; AI judging inherits model biases; principles → behavior not fully controllable; not the same as truly safe; helpfulness still relies on humans.