Cross-Entropy in machine learning

Module 01 · Machine Learning · 55 concepts

In depth / 15 lines

Cross-Entropy, in more detail

Cross-entropy is the loss function used for essentially every classifier.

The model outputs a probability for each class, and one of them is correct.

The loss depends only on the probability given to the correct one.

It is the negative logarithm of that probability, which sounds worse than it is.

The logarithm is what shapes the incentive.

Giving the right answer 0.9 costs very little.

Giving it 0.5 costs noticeably more.

Giving it 0.01 costs enormously, since the logarithm heads towards infinity as probability heads to zero.

So confident and wrong is punished far harder than unsure and wrong.

That is what pushes the model towards honest probabilities rather than bravado.

Binary cross-entropy is the two-class version and pairs with a sigmoid output.

Categorical cross-entropy handles many classes and pairs with a softmax output.

It is smooth and differentiable, which accuracy is not, so gradients exist everywhere.

It comes from information theory, where it measures the cost of using the wrong distribution.

In practice you rarely think about that and simply pair it with your softmax.

Want the full technical article? Wikipedia ↗

Deep dive

What is underneath Cross-Entropy

What it measures

Cross-entropy is the average number of bits you waste by encoding data drawn from one distribution using a code built for another.

For classification the true distribution puts all its mass on the correct class, and the other distribution is whatever the model predicted.

H(y, \hat{y}) = - \sum_{i=1}^{K} y_i \log \hat{y}_i

$y_i$ : the true probability of class $i$ , which is 1 for the right class and 0 for the rest.
$\hat{y}_i$ : what the model predicted for that class.
$K$ : the number of classes.

Why it collapses to one term

Because $y$ is one-hot, every term in that sum where $y_i$ is zero contributes nothing.

\mathcal{L} = -\log \hat{y}_c

$c$ : the index of the correct class.

So the loss for one example depends on exactly one number: the probability the model gave the right answer.

Predict 0.9 and the loss is 0.105. Predict 0.1 and it is 2.303. Predict 0.01 and it is 4.605.

The penalty grows without bound as the prediction approaches zero, which is what makes it punish confident mistakes so much harder than uncertain ones.

Over a whole batch

\mathcal{L} = -\frac{1}{N} \sum_{n=1}^{N} \log \hat{y}^{(n)}_{c_n}

The mean, not the sum, so the learning rate does not have to change when the batch size does.

The gradient, and why it is so clean

Composed with a softmax, the derivative of this loss with respect to the logits is not the matrix softmax alone would give you.

\frac{\partial \mathcal{L}}{\partial z_i} = \hat{y}_i - y_i

The error is simply the predicted probability minus the true one.

That is the reason the two are fused into a single operation in every framework: computing them separately means building a matrix that immediately cancels.

One example, start to finish

The network produces logits, one per class.
Softmax turns them into probabilities.
Look up the probability of the correct class.
Take its negative logarithm, and that is the loss.
Backward, the gradient at the logits is just prediction minus target.

The two things that go wrong

\mathcal{L} = -\log(\hat{y}_c + \epsilon)

A predicted probability of exactly zero makes the logarithm negative infinity, so implementations either add a small epsilon or work in log space throughout.

The second is passing already-softmaxed probabilities into a function that expects logits, which applies softmax twice and quietly flattens every gradient.

Test yourself

One question on this card. Every option explains itself once you pick, right or wrong.

Question 1

Which prediction does cross-entropy punish the hardest?

Machine Learning

55 concepts in this module, then a graded test with every answer explained.

Study this module →