Rewards

A reward is the numerical score a reinforcement learning agent is trying to maximise.

It arrives from the environment after an action, and it is the only training signal there is.

The agent is never told what to do, only how well things went.

Dense rewards arrive at nearly every step, such as points scored during a game.

Sparse rewards arrive rarely, such as a single win or loss at the very end.

Sparse rewards are harder because credit for the outcome must be spread over many actions.

That difficulty has a name, and it is the credit assignment problem.

Reward shaping adds intermediate rewards that mark progress towards the real goal.

Shaping speeds learning up but risks teaching the intermediate goal instead of the real one.

The discount factor decides how much a distant reward is worth against an immediate one.

A discount near zero makes the agent short-sighted, and one near one makes it patient.

Designing the reward is the part of reinforcement learning that most often goes wrong.

Any measurable proxy will be optimised exactly as written, and never as intended.

In RLHF the reward comes from a model trained on human rankings rather than from the world.

Whatever you reward is what you will get, so state the goal carefully.