RLHF (Reinforcement Learning from Human Feedback)
RLHF is a training method that aligns a model with human preferences by learning a reward model from human comparisons of outputs, then optimizing the model against that reward with reinforcement learning. It is a core technique for making models more helpful, honest, and harmless than pre-training alone produces.
Definition
RLHF (Reinforcement Learning from Human Feedback)After pre-training and supervised fine-tuning, a model still may not behave the way people want. RLHF collects human rankings of candidate responses, trains a reward model to predict those preferences, and then uses reinforcement learning — historically PPO — to steer the model toward higher-reward, more preferred behavior.
RLHF was central to models like InstructGPT and underlies much of the helpfulness of modern chat assistants. Related approaches such as Constitutional AI and direct preference optimization aim to reduce the human-labeling burden or simplify the optimization while pursuing the same alignment goal.
Frequently asked questions
What is RLHF (Reinforcement Learning from Human Feedback)?
RLHF is a training method that aligns a model with human preferences by learning a reward model from human comparisons of outputs, then optimizing the model against that reward with reinforcement learning. It is a core technique for making models more helpful, honest, and harmless than pre-training alone produces.