DPO in LLMs, explained in 5 lines

Module 16 · Advanced AI · 23 concepts

In depth / 15 lines

DPO, in more detail

DPO is Direct Preference Optimization, a simpler route to the same destination as RLHF.

RLHF has three stages: collect preferences, train a reward model, then run reinforcement learning.

The middle stage is a whole extra model, and the last is a fragile training loop.

DPO removes both by optimising the language model directly on the preference pairs.

The insight is that the optimal policy under the RLHF objective has a closed form.

So you can write a loss over preferred and rejected responses that reaches the same place.

Raise the likelihood of the preferred response and lower the rejected one.

A reference model keeps it from drifting too far, playing the role the KL penalty played.

It is ordinary supervised training, so it needs no sampling loop or reward model.

That makes it far cheaper, far more stable and much easier to get working.

It often matches RLHF quality on standard benchmarks.

It has become a common default for preference tuning, especially in open models.

RLHF retains an advantage where a reward model can score many fresh samples.

DPO learns only from the pairs it is given.

Variants such as IPO, KTO and ORPO adjust the loss for different data shapes.

Want the full technical article? Wikipedia ↗

Deep dive

What is underneath DPO

Removing the reward model

RLHF fits a reward model to preferences, then optimises a policy against it with reinforcement learning. DPO shows the middle step can be skipped.

\pi^{*}(y \mid x) = \frac{1}{Z(x)} \pi_{\mathrm{ref}}(y \mid x) \exp\left( \frac{1}{\beta} r(x, y) \right)

The KL-constrained objective has a closed-form solution: the optimal policy is the reference policy reweighted by the exponentiated reward.

r(x, y) = \beta \log \frac{\pi^{*}(y \mid x)}{\pi_{\mathrm{ref}}(y \mid x)} + \beta \log Z(x)

Rearranging it expresses the reward in terms of the policy, which means any policy already implies a reward function.

The loss

\mathcal{L}_{\mathrm{DPO}} = -\mathbb{E}\left[ \log \sigma\left( \beta \log \frac{\pi_{\theta}(y_w \mid x)}{\pi_{\mathrm{ref}}(y_w \mid x)} - \beta \log \frac{\pi_{\theta}(y_l \mid x)}{\pi_{\mathrm{ref}}(y_l \mid x)} \right) \right]

$y_w$ : the preferred response, $y_l$ the rejected one.
$\beta$ : the same KL strength as in RLHF, typically 0.1.

Substituting that expression into the Bradley-Terry preference likelihood makes the partition function $Z(x)$ cancel, because it appears in both terms of the difference.

What is left is a supervised loss on pairs, with no sampling, no reward model and no reinforcement learning.

\nabla \mathcal{L} \propto \sigma\big( \hat{r}_l - \hat{r}_w \big) \Big[ \nabla \log \pi_{\theta}(y_w) - \nabla \log \pi_{\theta}(y_l) \Big]

The gradient raises the likelihood of the preferred response and lowers the rejected one, weighted by how wrong the implicit reward currently is on that pair.

One training run, in order

Start from the supervised fine-tuned model, and keep a frozen copy as the reference.
Take a batch of prompts, each with a preferred and a rejected response.
Compute the log probabilities of both responses under both models.
Evaluate the loss and step the policy only.
Watch the implicit reward margin and the KL from the reference, which are the two diagnostics.

That is two models in memory, where RLHF with PPO needs four.

The reference can even be evaluated once and cached, since it never changes, which reduces the working set to one trainable model.

What it gives up

Learning from a fixed dataset makes DPO off-policy.

DPO learns from responses generated earlier, often by a different model, and never from what the current policy would say now.

RLHF samples fresh outputs and scores them, so it explores and can be corrected on its own current mistakes.

That difference matters most where the reward can be checked, such as code that runs or answers that verify.

\pi_{\theta}(y_l) \to 0 \text{ faster than } \pi_{\theta}(y_w) \to 1

A known failure is that the loss can be reduced mostly by suppressing the rejected response, so the likelihood of both falls and the model becomes less confident overall.

The variants address exactly that.

IPO changes the objective to avoid the degeneracy, KTO drops the requirement for pairs, and ORPO merges the preference term into the supervised stage so there is only one training run.

DPO is widely used for open models because it costs a fraction of RLHF to run.

How close it comes to online RLHF depends on the task, and it falls furthest behind where answers can be checked.

Test yourself

One question on this card. Every option explains itself once you pick, right or wrong.

Question 1

What does DPO remove compared with RLHF?

Advanced AI

23 concepts in this module, then a graded test with every answer explained.

Study this module →