RLHF with PPO, conceptually
Reinforcement Learning from Human Feedback (RLHF) is the classic post-training recipe behind helpful chat models. It teaches a model which outputs humans prefer, not just which output appears in a supervised dataset.
The three-stage shape
- SFT model — start with a model that already follows instructions.
- Reward model — train a model to score responses from human preference comparisons.
- PPO optimization — update the policy model to produce responses that score well.
The reward model becomes a proxy for human judgment. PPO then pushes the model toward answers the reward model likes while trying not to drift too far from the SFT model.
Why PPO needs constraints
If you optimize only reward, the policy can exploit reward-model bugs. RLHF usually adds a KL penalty so the new model stays near the reference model.
| Component | Purpose |
|---|---|
| Reward model | Approximate human preference |
| Policy model | The model being aligned |
| Reference model | Anchor to prevent wild drift |
| KL penalty | Discourage reward hacking |
Why teams often avoid it
RLHF with PPO is powerful but operationally heavy: preference data, reward-model training, policy optimization, stability tuning, and careful evaluation. For many fine-tuning projects, DPO gets most of the preference-alignment benefit with less machinery.
Pitfall
The reward model is not truth. It is another learned model with blind spots, and the policy will learn to exploit those blind spots if evaluation is weak.
Connects to: SFT · DPO · evaluating alignment