The loss function scores how far the network's output is from the truth.
Mean squared error is the standard choice for regression outputs.
Cross-entropy is the standard choice for classification outputs.
The loss is a single number computed over the whole batch.
Every gradient in the network is ultimately derived from this one number.
The loss function turns the gap between prediction and truth into one number.
Everything the network learns comes from trying to make that number smaller.
It is computed over a whole batch and averaged, not per example.
Mean squared error is the usual choice when the output is a quantity.
It squares the error, so being wrong by ten hurts a hundred times as much as being wrong by one.
That makes it sensitive to outliers, which is sometimes right and sometimes not.
Mean absolute error is the calmer alternative when a few extreme rows should not dominate.
Cross-entropy is the standard choice when the output is a class.
It looks at the probability given to the correct class and penalises confidence in the wrong one.
Being confidently wrong is punished far harder than being unsure, which is the behaviour you want.
Binary cross-entropy handles two classes and pairs with a sigmoid output.
Categorical cross-entropy handles many and pairs with a softmax output.
The loss has to be differentiable, or there is no gradient to work backwards from.
That is why accuracy is never used as a loss, since it moves in steps.
Pick the loss that matches your task, then watch it on validation data rather than training data.
One question on this card. Every option explains itself once you pick, right or wrong.
25 concepts in this module, then a graded test with every answer explained.