PPO in reinforcement learning, explained in 5 lines

Module 16 · Advanced AI · 23 concepts

In depth / 15 lines

PPO, in more detail

PPO is Proximal Policy Optimization, the workhorse of modern policy gradient methods.

The problem it solves is that a single large update can destroy a policy.

Policy gradients estimate a direction from sampled experience.

That estimate is only valid near the current policy.

Take too large a step and the data no longer describes the policy you now have.

The result is a collapse that training rarely recovers from.

PPO limits how far each update can move the policy.

It compares the new probability of an action against the old one as a ratio.

The objective is clipped so a ratio outside a small band gives no further benefit.

There is nothing to gain from a large step, so the optimiser does not take one.

The clipping is cheap, which is why PPO beat the more principled alternatives in practice.

It also reuses each batch of experience for several updates, improving sample efficiency.

It is stable enough to work across many problems with little tuning.

That reliability is why it became the algorithm behind most RLHF pipelines.

In that setting the reward comes from a learned reward model rather than the environment.

Want the full technical article? Wikipedia ↗

Deep dive

What is underneath PPO

The problem it fixes

\theta \leftarrow \theta + \alpha \nabla_{\theta} J

A plain policy gradient step can be too large. The policy changes sharply, the next batch is collected under a much worse policy, and the run never recovers.

r_t(\theta) = \frac{\pi_{\theta}(a_t \mid s_t)}{\pi_{\theta_{\mathrm{old}}}(a_t \mid s_t)}

The ratio between the new and old policy on the same action measures how far the update has moved, per sample.

\mathcal{L}^{\mathrm{CPI}} = \mathbb{E}\big[ r_t(\theta) \hat{A}_t \big]

Maximising that surrogate with no constraint pushes the ratio to infinity wherever the advantage is positive, which is the failure being prevented.

The clipped objective

\mathcal{L}^{\mathrm{CLIP}} = \mathbb{E}\Big[ \min\Big( r_t \hat{A}_t, \; \mathrm{clip}\big(r_t, \, 1 - \epsilon, \, 1 + \epsilon\big) \hat{A}_t \Big) \Big]

$\epsilon \approx 0.2$ : how far the ratio may move before the objective stops rewarding it.

When the advantage is positive, the objective stops improving once the ratio passes $1 + \epsilon$ , so there is no gradient to push further.

When the advantage is negative, the same applies below $1 - \epsilon$ .

\min(\cdot, \cdot)

The minimum is what makes it a pessimistic bound.

It takes the clipped value only when clipping makes the objective smaller, so a step that has already gone too far can still be corrected back.

Why it replaced TRPO

\mathbb{E}\big[ D_{\mathrm{KL}}\big(\pi_{\theta_{\mathrm{old}}} \Vert \pi_{\theta}\big) \big] \leq \delta

Trust region policy optimisation enforced a hard KL constraint, which required second-order information and a conjugate gradient solve at every update.

PPO gets most of the benefit with a first-order method and a clip, so it runs on ordinary optimisers and a few lines of code.

That simplicity is the whole reason it is the default, and it is why the same algorithm trains robots and language models.

One iteration, in order

Run the current policy and collect a batch of transitions.
Compute advantages, usually by generalised advantage estimation.
Take several gradient epochs over that same batch under the clipped objective.
Update the value network on the same data with a squared error loss.
Discard the batch and collect again with the new policy.

Step three is the practical gain. Clipping makes it safe to reuse a batch several times, where a plain policy gradient could use it once.

In RLHF

\mathcal{L} = \mathcal{L}^{\mathrm{CLIP}} - \beta \, D_{\mathrm{KL}}\big( \pi_{\theta} \Vert \pi_{\mathrm{ref}} \big)

Language model training adds an explicit penalty against the original supervised model, which is a different KL from the one PPO already controls.

The clip stops the update from being too large in one step. This term stops the policy from drifting far from a sensible model across many steps.

Without it the policy finds text that scores highly under the reward model and reads as nonsense, which is reward hacking in its clearest form.

\beta \approx 0.01 \text{ to } 0.1

Four models are in memory during this: the policy, the reference, the reward model and the critic, which is why direct preference methods that need only two became popular.

Test yourself

One question on this card. Every option explains itself once you pick, right or wrong.

Question 1

What problem does PPO's clipping solve?

Advanced AI

23 concepts in this module, then a graded test with every answer explained.

Study this module →