Reinforcement learning trains an agent through trial, error and reward.
The agent observes a state, takes an action and receives a reward.
Its goal is maximising total reward over time, not just the next step.
Exploration finds new strategies while exploitation uses known good ones.
Balancing those two is the central difficulty of RL.
Reinforcement learning trains an agent by letting it act and rewarding what works.
There is no dataset of correct answers, which is the fundamental difference from supervised learning.
The agent observes a state, chooses an action, and receives a reward and a new state.
It repeats that millions of times and gradually learns what to do.
The goal is the total reward over time, not the reward from the next step.
That distinction is the whole difficulty, since a good move now may cost later.
Credit assignment is the hard part: which of a hundred actions caused the eventual outcome?
A chess game is won or lost at the end, and every move shares the blame.
Exploration means trying things to find out whether they are better.
Exploitation means using what already works.
Too much exploitation locks in a mediocre strategy, and too much exploration never converges.
Balancing them is the central practical problem.
It is sample-inefficient, often needing millions of episodes to learn what a person learns in ten.
That is why it shines in simulation, where episodes are cheap and fast.
In language models it appears as the preference-tuning stage rather than as the main training.
The agent is not trying to be right about the next step. It is trying to maximise the total reward from here onward, which is the return.
Discounting makes the sum finite on an infinite horizon and expresses how much a reward later is worth compared with one now.
That is roughly the effective horizon: means the agent cares about the next hundred steps or so.
Nobody says which action was correct. The agent receives a number, and it has to work out which of its past actions earned it.
This is the credit assignment problem.
A reward at step 100 may be due to a choice at step 3, and separating that from the ninety-seven irrelevant choices is the central difficulty.
The data is also not fixed.
The agent's own policy decides what it sees next, so the distribution changes as it learns, which breaks the independence assumption every supervised method relies on.
The state value is how good a position is under the current policy. The action value is how good a particular move from it is.
Knowing gives the optimal policy immediately, which is why so many methods learn it instead of learning the policy.
Value-based methods learn and act greedily.
Q-learning and its deep variants estimate values and derive the policy from them, which suits discrete action sets.
Policy-based methods learn directly.
Policy gradients optimise the policy itself, which handles continuous actions and naturally represents a stochastic policy.
Actor-critic methods do both.
The actor chooses and the critic evaluates, which keeps the policy gradient's generality while using a learned value to reduce its variance.
PPO is in this family, and it is what trains language models from human feedback.
The practical constraint across all three is how many environment interactions are needed.
Millions is normal, which is why simulation dominates and why real-world reinforcement learning is hard.
One question on this card. Every option explains itself once you pick, right or wrong.
23 concepts in this module, then a graded test with every answer explained.