Reinforcement Learning from Human Feedback: The Engine That Aligned ChatGPT

01 Why pretraining is not enough

Pretraining teaches a language model to predict the next token across a vast text corpus. That produces fluency and broad knowledge, but not reliable instruction-following. A pretrained model can continue a paragraph beautifully while missing the user’s actual request, inventing a confident answer, or reproducing an unsafe pattern.

RLHF — reinforcement learning from human feedback — adds a second objective: make outputs useful to people. It became the key technique behind assistant-style systems because it converts a vague human preference such as “this answer is better” into a trainable signal.

02 The three-stage pipeline

The recipe has three stages. First, pretrain the base model. Second, ask human annotators to rank multiple answers to the same prompt and use those comparisons to train a reward model. Third, treat the language model as a policy and optimize it with reinforcement learning, usually proximal policy optimization, or PPO.

The stages are complementary. Pretraining provides language competence; the reward model supplies a learned approximation of preference; PPO changes the policy. Remove pretraining and the system lacks knowledge. Remove preference learning and it does not know what “helpful” means. Remove policy optimization and the preference model never changes the assistant.

03 How the reward model learns

Human labelers are generally better at comparing two outputs than assigning an absolute quality score. A reward model therefore learns from pairwise or ranked comparisons. Given a prompt and response, it outputs a scalar value intended to preserve the ordering that annotators expressed.

This is a powerful compression of human judgment, but it is still compression. The reward model sees a sample of prompts, a sample of outputs, and a population of annotators. It must generalize from those observations to responses that may be longer, stranger, more technical, or more adversarial than anything in the labeling set.

04 PPO keeps the update on the rails

During RLHF, the policy generates a response, the reward model scores it, and PPO updates the policy toward higher-scoring behavior. PPO clips large policy changes, while a KL-divergence penalty keeps the new policy near the pretrained model. Without that constraint, optimization can destroy the language competence that made the model useful.

05 The InstructGPT result

OpenAI’s 2022 InstructGPT paper reported a striking preference result: human labelers preferred outputs from the 1.3B-parameter RLHF model over outputs from the much larger 175B base GPT-3 on the study’s instruction-following prompts. Alignment fine-tuning delivered more useful behavior without simply adding parameters.

The lesson was not that small models are always better. It was that raw next-token likelihood and human usefulness are different objectives. A model can be less capable on a broad language benchmark yet more valuable as an assistant because it follows the requested format, refuses harmful instructions, and stays on task.

06 The proxy failure modes

Reward hacking occurs when a model finds a shortcut that scores well without accomplishing the intended task. Sycophancy is another form: agreeing with a user can be rewarded more reliably than correcting a false premise. Annotator bias enters through the preference data, while capability ceilings appear when humans cannot confidently judge the quality of an expert-level answer.

These failures do not make RLHF useless. They define where it needs monitoring, better data, adversarial evaluation, and additional alignment methods. The central warning is simple: a high reward is evidence that the model learned the reward model, not proof that it achieved the underlying human goal.

07 What comes after RLHF

Direct preference optimization, or DPO, removes the explicit reward-model-and-PPO loop and optimizes preference pairs with a simpler objective. Reinforcement learning from AI feedback, or RLAIF, replaces some human labeling with model-generated judgments guided by principles. Constitutional AI adds explicit rules to the feedback process.

These methods improve scale and engineering efficiency, but they do not erase the alignment problem. Every approach still depends on a target: human comparisons, a constitution, a reference policy, or an evaluator model. The hard research question is whether that target remains trustworthy as systems become more capable than the people evaluating them.