RLHF in LLMs, explained in 5 lines

Module 16 · Advanced AI · 23 concepts

In depth / 15 lines

RLHF, in more detail

RLHF is reinforcement learning from human feedback.

It exists because the thing we want cannot be written as a loss function.

There is no formula for helpful, harmless and honest.

People can compare two responses and say which is better, even when they cannot define better.

So the process starts by collecting those comparisons.

Humans rank several responses to the same prompt from best to worst.

A reward model is trained to predict those preferences from the text alone.

That turns scattered human judgement into a function that can score any output.

The language model is then optimised against that reward model, usually with PPO.

A penalty keeps it from straying too far from the original model.

Without that penalty it drifts into strange text that happens to score well.

This is the stage that turned capable base models into usable assistants.

It is also where refusal behaviour and tone are shaped.

It has costs, including a documented tendency towards agreeableness.

The reward model is a proxy, so optimising it hard invites reward hacking.

Want the full technical article? Wikipedia ↗

Deep dive

What is underneath RLHF

Learning a reward nobody can write down

Helpfulness has no formula. What exists instead is the ability to say, of two answers, which one is better.

P(y_1 \succ y_2 \mid x) = \frac{\exp\big(r(x, y_1)\big)}{\exp\big(r(x, y_1)\big) + \exp\big(r(x, y_2)\big)}

The Bradley-Terry model turns a preference into a probability, so a set of comparisons becomes a likelihood and a reward function can be fitted to it.

\mathcal{L}_{r} = -\mathbb{E}\Big[ \log \sigma\big( r(x, y_w) - r(x, y_l) \big) \Big]

$y_w$ : the preferred answer, $y_l$ the rejected one.

Only the difference appears, so the reward is defined up to an additive constant, which is fine because only differences are ever used.

The three stages

Supervised fine-tuning, to get a model that answers in the right shape at all.
Reward modelling, on human comparisons of that model's own outputs.
Reinforcement learning against the reward model, with PPO.

\max_{\pi} \; \mathbb{E}\big[ r(x, y) \big] - \beta \, D_{\mathrm{KL}}\big( \pi \Vert \pi_{\mathrm{SFT}} \big)

Stage three maximises the learned reward while penalising drift from the supervised model.

Comparisons are collected on the model's own outputs rather than on written ideals, which is why this needs stage one before it can start.

Why the KL penalty is load-bearing

r_{\mathrm{total}} = r_{\phi}(x, y) - \beta \log \frac{\pi_{\theta}(y \mid x)}{\pi_{\mathrm{SFT}}(y \mid x)}

The reward model is a model. It is accurate near the distribution it was trained on and arbitrary outside it.

Without the penalty, the policy finds text that scores extremely well and is not language, because nothing constrains it to stay where the reward model is valid.

If $\beta$ is too high nothing is learned, and if it is too low the policy hacks the reward.

That single coefficient is the most consequential hyperparameter in the pipeline.

What the reward model is worth

\text{agreement with humans} \approx 70 \text{ to } 75\%

Preference labels are noisy, since two annotators disagree about a quarter of the time, and the reward model inherits that ceiling.

Overoptimisation is when true quality falls while $r$ keeps rising.

Beyond a certain amount of optimisation, the policy is exploiting the reward model's errors rather than satisfying the preference it approximates.

Length is the best documented example. Preference data favours longer answers, so reward models learn to reward length, and policies learn to pad.

The memory bill, and what replaced it

\pi_{\theta}, \; \pi_{\mathrm{ref}}, \; r_{\phi}, \; V_{\psi}

Four models are loaded at once during stage three, which makes RLHF the most expensive part of building an assistant.

\mathcal{L}_{\mathrm{DPO}} = -\log \sigma\left( \beta \log \frac{\pi_{\theta}(y_w)}{\pi_{\mathrm{ref}}(y_w)} - \beta \log \frac{\pi_{\theta}(y_l)}{\pi_{\mathrm{ref}}(y_l)} \right)

Direct preference optimisation shows the optimal policy can be written in closed form in terms of the reward, which lets the reward model be eliminated entirely.

Two models, a supervised-style loss, no sampling loop, and results close to full RLHF, which is why most open models now use it.

RLHF retains an advantage where the reward can be checked rather than preferred.

Online sampling lets the policy explore and be scored on outcomes a fixed preference set never contained.

Test yourself

One question on this card. Every option explains itself once you pick, right or wrong.

Question 1

What does DPO remove compared to RLHF?

Advanced AI

23 concepts in this module, then a graded test with every answer explained.

Study this module →