Gradient descent updates every weight using its computed gradient.
Full batch descent uses the entire dataset for a single update.
That is accurate but far too slow and memory-hungry for real datasets.
The learning rate scales how far each weight moves.
Too high and it diverges, too low and it crawls.
Gradient descent is the rule that turns gradients into an actual change in the weights.
Every weight moves a small step in the direction that lowers the loss.
How far it moves is its gradient multiplied by the learning rate.
So a weight the loss is sensitive to moves further than one it barely notices.
Full batch descent computes the gradient over the entire training set before moving.
That gives the truest direction available and is almost never used.
One update would need a pass over millions of rows, and you need thousands of updates.
The picture to hold is a landscape where height is the loss and position is the weights.
Descent feels the slope underfoot and steps downhill without seeing the whole map.
The learning rate is the length of that step.
Too long and you stride across the valley and end up higher than you started.
Too short and you creep, needing a hundred times more steps than you have patience for.
The landscape has many valleys, and you are not promised the deepest one.
In practice the good valleys in a large network are close enough in quality to not matter.
Everything else in training is a smarter way to choose the direction and the step size.
One question on this card. Every option explains itself once you pick, right or wrong.
25 concepts in this module, then a graded test with every answer explained.