Constitutional AI: Harmlessness from AI Feedback

Constitutional AI asks whether a short, explicit set of principles can replace repeated human harmlessness labels without reducing safety to blanket refusal. Read it to see how one rulebook produces self-revised demonstrations and AI-labelled response preferences, why the policy still depends on human helpfulness data and evaluation, and how inaccurate critiques, judge capability, principle choice, and reward over-optimization bound the result.

Reading focus: How sampled principles turn a model's first response into critiques, revisions, and supervised training targets. How an AI feedback model converts the same kind of principles into soft pairwise labels for a preference model and reinforcement learning. Why removing direct human harmlessness labels does not remove human values, evaluation, red-team prompts, or reward-model failure modes.

Anthropic technical report, 2022. Yuntao Bai et al.. 35 min read, medium difficulty.