Optimization means finding parameter values that make the loss as small as possible.
Picture the loss as a landscape of hills and valleys.
You want to walk downhill toward a low point.
Algorithms like gradient descent decide which direction downhill is.
Optimization is one of the mathematical foundations that all of ML sits on.
Optimization is the search for the settings that make the loss smallest.
Picture a landscape where height is error and your position is the parameters.
You are looking for the lowest valley, in the dark, one step at a time.
For a simple model that landscape is a smooth bowl with one bottom.
For a neural network it is a mountain range full of dips.
You will not find the lowest point in that range, and you do not need to.
Many of the dips turn out to be about as good as each other.
The step size decides whether you settle or bounce around forever.
Where you start matters, which is why weights begin random instead of at zero.
Modern methods keep a memory of recent steps, which smooths the path.
They also give every parameter its own step size.
Training that refuses to settle is usually a step size problem first.
A loss that turns into nothing at all usually means the numbers blew up.
Watching the loss curve tells you more than any single final number.
Optimization finds a good fit, not a correct answer.
One question on this card. Every option explains itself once you pick, right or wrong.
55 concepts in this module, then a graded test with every answer explained.