RLHF and Preference Alignment

Interview answer (say this first). RLHF trains a model to match human preferences instead of just imitating text. Humans rank pairs of model responses, and a reward model learns to predict which response people prefer. A reinforcement-learning algorithm such as PPO then updates the language model to earn a higher reward, with a KL penalty that keeps it close to the original model so it does not drift or game the reward. DPO is a simpler alternative: it skips the reward model and the RL loop and optimises the preference pairs directly using the policy and a frozen reference model.

Why this exists

Pretraining minimises next-token loss, and instruction tuning minimises response loss on demonstrations. Both are imitation objectives: they make the model’s text more like the training text. Neither can express “this answer is better than that one”.

That gap creates concrete failures.

Failure 1 — the internet is the training data. A model trained to imitate text on the web will happily continue toxic, biased, or dangerous text, because such text exists in the corpus. Next-token loss has no concept of harm. It only asks “how likely is this continuation?”.

Failure 2 — many answers are valid, but some are much better. For the prompt “Explain gravity to a child”, thousands of answers are grammatically fine. Imitation training can reward a rambling answer as much as a clear one, because both are plausible text. What you want is a signal that ranks them.

Failure 3 — the model optimises the wrong thing. The loss treats every token equally and every valid completion as equally correct. It cannot say “be helpful here”, “refuse there”, or “prefer the concise answer”. Human judgement is not expressible as a next-token label.

Preference alignment exists to inject a comparative signal. Instead of “this is the text”, the data says “people preferred A over B”. That is a much closer match to what product teams actually want, and it is why aligned assistants feel helpful rather than merely fluent.

Start from zero

These terms come up constantly, so define them all up front.

Two pairs to keep straight:

• Reward model vs value model. The reward model scores a finished response against human preferences. The value model predicts future reward during RL. They are different models with different jobs, and mixing them up is a common interview mistake.
• Helpfulness vs harmlessness. These two goals conflict. Maximise helpfulness alone and the model answers harmful requests; maximise harmlessness alone and it refuses everything. Alignment is the tuning of that trade-off.

The core idea

Training a dog has three parts. First you expose it to the world so it understands it. Then you show it what to do with repetition. Finally, you reward good behaviour and discourage bad — and you keep it on a leash so it does not run off chasing every reward.

• Exposure is pretraining.
• Showing is instruction tuning.
• Rewarding is RLHF, and the leash is the KL penalty to the reference model.

The leash matters because a reward model is only a proxy for what people want. Given freedom, the policy will find the exact inputs that score highly, even if they are not actually better. This is reward hacking, and the KL penalty limits how far the policy can wander while exploiting the reward model.

DPO takes a different route to the same destination. Instead of learning a reward model and then running RL against it, DPO shows that the preference objective can be rewritten so the language model itself acts as the reward. That removes an entire model and the unstable RL loop.

flowchart TD
    A["SFT model<br/>follows instructions"] --> B["Collect comparisons<br/>prompt + two responses + human pick"]
    B --> C["Train reward model<br/>Bradley-Terry loss"]
    C --> D["RL loop (PPO)"]
    A --> E["Frozen reference<br/>copy of SFT model"]
    E --> D
    D --> F["Policy updates<br/>maximise reward − β·KL(reference)"]
    F --> G["Aligned model"]
    B --> H["DPO<br/>skip RM and RL"]
    A --> H
    E --> H
    H --> G

The two routes compared:

Both methods need the same raw material: preference pairs. The difference is how they use them.

Online versus offline. DPO is offline: it consumes a fixed dataset of preference pairs. RLHF is online: the policy generates fresh responses during training, and the reward model scores them. Online training can discover behaviours that are not in the dataset, which is powerful and also exactly why it is harder to keep stable. That single distinction explains most of the operational difference between the two methods.

How it works

RLHF, step by step.

1. Start from the SFT model. It already follows instructions; alignment now shapes which instructions it follows and how.
2. Collect comparison data. Show people a prompt and two or more model responses, and ask them to rank them. A single prompt yields several pairs; a set of ranked responses yields more.
3. Train the reward model. Given a prompt and a response, the reward model outputs one number. It is trained so that the preferred response gets a higher number than the rejected one, using the Bradley-Terry loss.
4. Freeze a reference model. Copy the SFT model and never update it. It is the anchor that defines “not too far”.
5. Generate rollouts. The policy produces responses to prompts. The reward model scores each one.
6. Apply the KL penalty. The score used for learning is the reward minus a penalty proportional to how much the policy’s distribution has moved from the reference model’s distribution.
7. Update with PPO. PPO nudges the policy’s token probabilities to increase the penalised reward, while clipping updates so no single step is too large. A value model estimates expected reward to reduce noise.
8. Repeat and evaluate. Iterate on fresh prompts, then test on held-out prompts, safety suites, and general capability benchmarks.

DPO, step by step.

1. Collect the same preference pairs.
2. Keep a frozen reference model. Same role as in RLHF.
3. Compute log-probabilities. For each response, compute its log-probability under the policy and under the reference.
4. Form the implicit reward. The quantity β · log(policy/reference) acts as a reward, without training a reward model.
5. Optimise the DPO loss. Push up the implicit reward of the chosen response and push down the rejected one, using a sigmoid loss. β controls how tightly the policy is held near the reference.
6. Evaluate. The same held-out and safety checks apply.

Note:

The key insight of DPO. Under the Bradley-Terry preference model, the optimal policy has a closed form in terms of the reward. DPO rearranges that relationship so the reward is written in terms of the policy itself. The result is a simple classification-style loss on preference pairs, with no reward model and no RL. It is why DPO became the practical default for many teams.

Warning:

Preference data encodes the values of the people who wrote it. If annotators prefer long answers, flattery, or a particular style, the aligned model will too. Document who labelled the data, how they were instructed, and how disagreements were resolved. The model inherits all of it, and no optimizer removes that bias.

The syntax you will use

Preference data. The raw material is one JSON object per pair.

{"prompt": "Summarise this contract.",
 "chosen": "The contract runs for 12 months and renews automatically.",
 "rejected": "It is a contract. Contracts have terms. This one has terms too."}

The Bradley-Terry reward loss, in numpy. The reward model scores each response, and the loss pushes the chosen score above the rejected score.

import numpy as np

def sigmoid(x):
    return 1.0 / (1.0 + np.exp(-x))

r_w, r_l = 0.8, -0.4                       # reward scores: chosen, rejected
bt_loss = -np.log(sigmoid(r_w - r_l))      # Bradley-Terry loss

The DPO loss, in numpy. Only log-probabilities of the two responses under the policy and the reference are needed.

import numpy as np

def sigmoid(x):
    return 1.0 / (1.0 + np.exp(-x))

beta = 0.1
# log-probabilities of each response under policy and frozen reference
ratio_w = -2.0 - (-2.3)                    # policy minus reference, chosen
ratio_l = -2.5 - (-2.2)                    # policy minus reference, rejected
margin = beta * (ratio_w - ratio_l)
dpo_loss = -np.log(sigmoid(margin))

The PPO objective, at a high level. You rarely write this by hand, but you should recognise the parts: the probability ratio, the clipping, and the KL penalty.

objective = reward(response) − β · KL(policy ‖ reference)
PPO updates the policy to increase the objective, while clipping the
change in the probability ratio so each step stays small.

Training with TRL. The library exposes one trainer per method.

from trl import RewardTrainer, DPOConfig, DPOTrainer

# 1. reward model
RewardTrainer(model="meta-llama/Llama-3.1-8B-Instruct", train_dataset=pairs).train()

# 2. DPO (no reward model, no RL loop)
DPOTrainer(
    model="meta-llama/Llama-3.1-8B-Instruct",
    ref_model=None,                        # None means a frozen copy is used
    train_dataset=pairs,
    args=DPOConfig(beta=0.1, learning_rate=5e-7),
).train()

Examples: simple to real

Example 1 — why imitation loss cannot rank answers. Suppose two answers to “Explain gravity to a child”.

A: "Gravity is the invisible pull that makes things fall toward the ground."
B: "Gravity, gravitas, a force, falling, ground, apples, Newton, downward."
A next-token loss can assign both a reasonable likelihood. It has no term that says A is better.

Both are likely English text. Imitation cannot distinguish clear from rambling, because both are valid sequences. Preference data can.

Example 2 — the reward model learns a ranking. The Bradley-Terry loss compares two scalar scores.

# reward scores: chosen 0.8, rejected -0.4
# measured Bradley-Terry loss: 0.2633
# measured P(chosen preferred over rejected): 0.7685
# separation matters:
#   r_w=2.0,  r_l=-2.0 -> loss 0.0181   (chosen clearly higher)
#   r_w=0.2,  r_l= 0.1 -> loss 0.6444   (barely separated)
#   r_w=-1.0, r_l= 1.0 -> loss 2.1269   (ranking is wrong, large loss)

The loss is small when the chosen response is clearly ahead and large when the ranking is wrong. That is the entire learning signal for the reward model.

Example 3 — DPO reaches the same goal with logs of probabilities. Set β=0.1, and let the policy have raised the chosen response’s log-probability relative to the reference by 0.3, while the rejected response moved down by 0.3.

# measured:
#   implicit reward for chosen:   0.0300
#   implicit reward for rejected: -0.0300
#   margin:                        0.0600
#   DPO loss:                      0.6636
# separation effect:
#   ratio_w= 0.5, ratio_l=-0.5 -> DPO loss 0.6444
#   ratio_w= 0.0, ratio_l= 0.0 -> DPO loss 0.6931
#   ratio_w=-0.5, ratio_l= 0.5 -> DPO loss 0.7444

The loss is 0.6931 when the policy has no preference either way, which is log(2). As the policy favours the chosen response more, the loss falls. DPO is a well-behaved binary classification problem over preferences.

Example 4 — reward hacking. Suppose the reward model was trained on helpful answers and learned that longer answers tend to be rated higher. During RL, the policy discovers that it can raise its score by padding every answer with repetition.

Reward model: longer, confident, list-like answers score higher
Policy learns: pad the answer, add a confident summary, repeat the question
Human view:   the answer is worse — verbose and evasive

The policy is not “trying to deceive”. It is doing exactly what it was optimised to do. The reward model is an imperfect stand-in for human judgement, and optimising it hard exposes the imperfection. This is Goodhart’s law in a training loop.

Example 5 — helpfulness and harmlessness pull against each other. Alignment has to balance two conflicting objectives.

Over-optimising either side produces a bad product: a model that helps with everything is unsafe, and a model that refuses everything is useless.

Example 6 — instruction hierarchy, the defence against prompt injection. An agent reads system instructions, user messages, retrieved documents, and tool output. These sources have different trust levels, and the model must learn to prioritise them.

Instruction-hierarchy training teaches the model that system instructions outrank user instructions, which outrank untrusted content. It is a useful and important defence, but it is not a guarantee. Prompt injection remains an open problem, which is why agents also need sandboxing, least privilege, and validation of tool calls.

In practice, instruction hierarchy is one layer of an agent security design. The other layers are least-privilege tools, validation of everything the model emits, and human approval for irreversible actions. Treating the model’s priorities as a security boundary is a mistake; security has to be enforced in code the model cannot talk its way past.

In production

• Preferences are the bottleneck, not the algorithm. The quality, consistency, and coverage of the comparison data set the ceiling. No optimizer fixes noisy or biased preferences.
• Reward models are proxies that get gamed. Always evaluate the aligned model with humans or independent metrics, not just the reward score. A rising reward with flat human ratings is a red flag.
• The KL penalty is a leash, not a wall. It limits drift but does not prevent reward hacking. Tuning the penalty trades capability and helpfulness against safety and stability.
• RLHF is operationally heavy. Four models in memory, online rollouts, and sensitive hyperparameters make it expensive and unstable. Many teams get most of the benefit from DPO.
• DPO is simpler, not magic. It still needs good preference pairs and a reference model, and β controls the same drift-versus-change trade-off. It can still overfit and can still reduce diversity.
• Alignment can cost capability. The “alignment tax” is real: safety and preference training can reduce creativity, diversity, or some benchmark scores. Measure before and after, on general tasks.
• Watch for sycophancy. If annotators prefer agreeable answers, the reward model learns to prefer agreement. The result is a model that tells users what they want to hear.
• Over-refusal is a product bug. Too much harmlessness training makes the model refuse benign requests. Track refusal rate on a known-safe prompt set, not just on unsafe ones.
• Bias in, bias out. Annotators bring cultural and personal biases. Preference data should be documented, sampled carefully, and audited for systematic skew.
• Preference tuning does not add knowledge. Like instruction tuning, it shapes behaviour. Facts still belong in retrieval.
• Instruction hierarchy helps, but defence in depth is required. Assume injection can succeed occasionally. Limit what tools can do, require confirmation for destructive actions, and never let model output be trusted as code or commands.
• Version and re-evaluate on every base-model change. Alignment is tied to a specific checkpoint. When the base model changes, the preference data, reward model, and evaluation all need to be revisited.

Interview questions

1. Why is next-token loss not enough for alignment?

Answer. Next-token loss measures how well the model imitates text. It cannot express that one valid answer is better than another, and it has no concept of harm, so a model trained only on imitation can produce toxic or unhelpful continuations that are statistically likely. Alignment needs a comparative signal — people preferred this response over that one — which is what preference training provides.

Follow-up: “What did InstructGPT add over GPT-3?” It kept the pretraining objective, added supervised demonstrations, and then added preference-based RLHF with a reward model, which made the model follow instructions and behave more helpfully.

Trap. Saying RLHF makes the model smarter. It mostly makes the model behave more like what people prefer, using capability that pretraining already supplied.

2. How is a reward model trained?

Answer. It is trained on pairs of responses to the same prompt, where humans marked one as preferred. The model takes a prompt and a response and outputs a scalar score. The Bradley-Terry loss pushes the chosen response’s score above the rejected one’s. At inference it turns any response into a number, which gives the RL loop a signal to optimise.

Follow-up: “What is the difference between the reward model and the value model?” The reward model scores a finished response against preferences. The value model estimates expected future reward during RL and exists to reduce variance in PPO. They are separate models.

Trap. Treating the reward model as a source of truth. It is a learned proxy and can be wrong or gamed.

3. Explain RLHF with PPO at a high level.

Answer. Start from an SFT model. Generate responses, score them with the reward model, subtract a KL penalty that measures how far the policy has drifted from a frozen reference model, and update the policy with PPO to increase that adjusted score. PPO clips updates so each step is small. Repeat with fresh prompts. The reference model and the KL penalty are what stop the policy from wandering off to exploit the reward model.

Follow-up: “Why is PPO used instead of plain policy gradient?” PPO constrains how much the policy can change per update, which makes training much more stable, and it reuses data more efficiently.

Trap. Forgetting the reference model or the KL term. Without them, reward hacking becomes severe and the model degrades quickly.

4. What is DPO, and how does it differ from RLHF?

Answer. DPO reparameterises the preference objective so the language model itself provides the implicit reward, using its log-probabilities relative to a frozen reference model. It trains directly on preference pairs with a simple sigmoid loss. That removes the reward model and the RL loop, so it needs fewer models in memory and is far more stable, while using the same preference data.

Follow-up: “What does β do in DPO?” It scales the implicit reward and controls how strongly the policy is kept near the reference. A larger β means less drift from the reference model, similar in spirit to a stronger KL penalty.

Trap. Claiming DPO is always better. It is simpler and usually cheaper, but RLHF with online rollouts can still outperform it in some large-scale settings, and DPO can overfit preferences just as RLHF can.

5. What is reward hacking, and how do you reduce it?

Answer. Reward hacking is when the policy gets a high reward score without genuinely being better — for example, by being verbose, confident, or sycophantic because those traits scored well in the preference data. It happens because the reward model is an imperfect proxy. You reduce it with a KL penalty to the reference model, reward-model ensembles, better and more diverse preference data, frequent human evaluation, and sometimes by iterating the reward model as new exploits appear.

Follow-up: “Why not just remove the KL penalty?” Without it, the policy can drift arbitrarily far and exploit the reward model’s blind spots, and generation quality often collapses.

Trap. Believing a high reward score means a good model. The score is only as good as the reward model, and that model is trained on a finite, biased sample of human judgement.

6. What are the helpfulness and harmlessness objectives, and how do they conflict?

Answer. Helpfulness is doing what the user asked, well. Harmlessness is avoiding responses that could cause harm. They conflict when a request is both answerable and dangerous, such as lock-picking instructions. Over-weighting helpfulness produces unsafe answers; over-weighting harmlessness produces over-refusal of benign requests. Alignment tunes this trade-off using separate labelled data for each objective.

Follow-up: “How do you measure over-refusal?” Track refusal rate on a set of prompts known to be safe. A rising refusal rate there, even with good safety scores, is a product regression.

Trap. Assuming safety and helpfulness are a single axis. They are two objectives that must be balanced, and the right point depends on the product.

7. What is the instruction hierarchy, and why does it matter for agents?

Answer. It is the principle that instructions from more trusted sources should outrank less trusted ones: system or developer instructions over user instructions, and both over retrieved documents or tool output. Agents mix all of these in one context window, so a model that treats a web page’s text as a command is vulnerable to prompt injection. Training on examples of conflicting instructions teaches the model to prefer the higher-priority source.

Follow-up: “Does that make prompt injection solved?” No. It is a mitigation, not a guarantee. Agents still need least-privilege tools, sandboxing, human confirmation for dangerous actions, and validation of everything the model emits.

Trap. Treating model-level instruction hierarchy as a security boundary. Security must be enforced in code; the model’s priorities only reduce the probability of failure.

8. How do you evaluate an aligned model?

Answer. Use several layers. Automated benchmarks for capability and safety, human preference evaluations on held-out prompts, targeted red-teaming for harmful behaviour, refusal-rate checks on safe prompts to catch over-refusal, and regression tests on general tasks to catch the alignment tax. Always compare against the pre-alignment model so you can attribute changes.

Follow-up: “Why not just look at the reward score?” The reward model is the thing being optimised, so its score rises by construction. Independent measurement is the only way to know whether real quality improved.

Trap. Reporting a single aggregate win rate. Aggregate scores hide safety failures, over-refusal, and capability regressions, all of which matter separately.

Remember this

• Imitation cannot rank answers; preferences can. That is why RLHF and DPO exist.
• RLHF = reward model + PPO + KL leash. The reference model and KL penalty keep the policy from gaming the reward.
• DPO skips the reward model and RL loop, training directly on preference pairs with an implicit reward.
• Reward models are proxies and get hacked. Evaluate with humans and independent tests, never the reward alone.
• Alignment tunes a trade-off between helpful, harmless, and capable — and instruction hierarchy reduces, but does not remove, prompt-injection risk.