RLHF (Reinforcement Learning from Human Feedback) is the technique most responsible for turning capable pretrained-and-fine-tuned LLMs into genuinely well-behaved assistants — using human preference comparisons to further train the model beyond what SFT's direct imitation alone can achieve.
The Three-Stage Process
- Start from an SFT model — the already fine-tuned model from Supervised Fine-Tuning serves as the starting point.
- Train a reward model: humans are shown multiple model-generated responses to the same prompt and asked to rank them by preference. A separate neural network (the reward model) is trained to predict these human preference rankings.
- Fine-tune the LLM via reinforcement learning: using the reward model as an automated feedback signal (standing in for actual human raters at scale), the LLM's own generation policy is further trained to produce outputs the reward model scores highly, typically via the PPO algorithm.
Reward Model Training — The Preference Loss
\(r(x,y)\) is the reward model's scalar score for response \(y\) given prompt \(x\). This is the Bradley-Terry preference model — the loss pushes the reward model to score the human-preferred response (\(y_{\text{chosen}}\)) higher than the rejected one, using exactly the sigmoid/logistic structure from Sigmoid Function.
The RL Fine-Tuning Objective — With a Crucial Safeguard
The model is trained to maximize reward — but with a KL divergence penalty (see KL Divergence) discouraging it from drifting too far from the original SFT model's behavior. Without this penalty, the model could learn to exploit weaknesses in the reward model (a failure mode called "reward hacking") — producing outputs that score artificially highly according to the reward model without actually being genuinely better responses.
Diagram
RLHF's three stages, each building on the previous one's output.
Common Mistakes
- Forgetting the KL penalty's role — without it, RL fine-tuning can drift into "reward hacking," producing outputs that game the reward model's specific weaknesses rather than genuinely improving response quality.
- Assuming RLHF trains directly on raw human ratings — it trains a separate reward model first, which then stands in for scalable, automated feedback during the RL stage, since collecting fresh human feedback for every single RL training step would be far too slow and expensive.
Interview Relevance
Q: "Why does RLHF fine-tuning include a KL divergence penalty against the original SFT model?" Without it, the RL optimization process can learn to exploit specific weaknesses or blind spots in the reward model — producing outputs that score artificially high according to the reward model without actually being better responses, a failure mode called reward hacking. The KL penalty constrains the model to stay reasonably close to its original SFT behavior, limiting how far this exploitation can drift.
Practice Question
Why is a separate reward model trained, rather than directly using live human ratings as the reinforcement learning signal at every training step?
