A policy is the rule an agent uses to choose an action in a given state.
It is the thing being learned, and the finished product of training.
A deterministic policy always picks the same action for the same state.
A stochastic policy samples from a probability distribution over actions.
Value methods arrive at a policy indirectly; policy methods learn it head-on.
A policy is the rule an agent uses to choose an action from the state it observes.
It is written as pi, and it is the object that reinforcement learning is trying to learn.
Everything else in the setup exists either to improve it or to measure it.
A deterministic policy returns one action for a given state, every time.
A stochastic policy returns a probability distribution over actions and samples from it.
Stochastic policies explore naturally, because the randomness is built into the behaviour.
They are also easier to differentiate, which is what policy gradient methods need.
Value-based methods such as Q-learning never represent the policy directly.
They learn the value of actions and then act greedily with respect to those values.
Policy-based methods parameterise the policy itself and adjust it with gradients.
Actor-critic methods keep both, using a learned value estimate to steady the policy update.
On-policy methods learn only from data the current policy generated.
Off-policy methods can learn from old or unrelated behaviour, which is far more data-efficient.
In RLHF the language model is the policy, and each token it emits is an action.
When training finishes, the policy is the thing you keep and deploy.
One question on this card. Every option explains itself once you pick, right or wrong.
23 concepts in this module, then a graded test with every answer explained.