A gradient is the derivative of the loss with respect to a parameter.
Its sign says which direction increases the loss.
Its magnitude says how sensitive the loss is to that parameter.
Moving opposite to the gradient is what reduces the error.
Gradients that shrink toward zero or blow up are a core training problem.
A gradient is the slope of the loss with respect to one parameter.
It answers what happens to the error if this one weight increases slightly.
The sign tells you the direction, and the size tells you how sharply it responds.
A positive gradient means increasing that weight makes the loss worse.
So you move the weight the other way, which is where gradient descent gets its name.
A gradient near zero means the loss barely notices this weight at all.
Every parameter has its own gradient, so a model with a million weights has a million of them.
Collected together they point in the direction that increases the loss fastest.
Stepping against that direction is the whole training algorithm.
Gradients are computed on a batch, so they are noisy estimates of the true slope.
That noise is not purely a problem, since it helps escape poor spots.
In deep networks gradients are products of many terms, layer after layer.
Products of small numbers vanish and products of large ones explode.
Watching the gradient size during training is the standard way to catch both.
Nearly every trick in deep learning exists to keep gradients in a workable range.
One question on this card. Every option explains itself once you pick, right or wrong.
25 concepts in this module, then a graded test with every answer explained.