An activation function adds non-linearity after a neuron's weighted sum.
Without one, stacking layers would collapse into a single linear function.
ReLU outputs the input if positive and zero otherwise, and is the usual default.
Sigmoid squashes into 0 to 1, and softmax turns a vector into probabilities.
The choice affects both how fast and how well the network trains.
An activation function is applied to a neuron's weighted sum before it is passed on.
Its job is to bend the straight line the neuron would otherwise draw.
Stack two linear layers and the result is still linear, so depth would buy nothing.
One bend per layer is what makes deep networks able to model complicated things.
ReLU passes positive numbers through unchanged and turns negatives into zero.
That is almost free to compute and its gradient does not shrink for positive inputs.
Its weakness is that a neuron stuck outputting zero receives no gradient and can die.
Leaky ReLU lets a small negative slope through so the neuron keeps a way back.
GELU is a smoother version of the same shape and is standard in transformers.
Sigmoid squashes everything into the range zero to one.
It saturates at both ends, where the gradient is nearly zero and learning stops.
That saturation is why sigmoid fell out of use inside deep networks.
It survives at the output layer of a binary classifier, where a probability is wanted.
Softmax is the multi-class version, turning a row of scores into probabilities that sum to one.
Inside the network use ReLU or GELU, and choose the output activation to match the task.
What the forward pass sees is the function. What training sees is the derivative, and that is what decides whether a network learns.
The sigmoid squashes to the range 0 to 1. Its derivative peaks at 0.25 and falls towards zero in both directions.
Tanh is the same shape centred on zero, with a peak derivative of 1, which is why it beat sigmoid in hidden layers for years.
ReLU is not a squash at all. Its derivative is exactly 1 on the positive side, so the gradient passes through unchanged however deep the stack is.
GELU multiplies the input by the probability that a standard normal is below it, giving a smooth curve that is ReLU-like for large values and gently negative near zero.
Backpropagation multiplies one derivative per layer, so a value below 1 at every layer shrinks the product geometrically.
With sigmoid at its best, ten layers multiply ten factors of at most 0.25, which is before any weight is considered.
With ReLU those factors are 1 wherever the unit is active, and the product does not decay at all. That is the whole reason deep networks became trainable.
A unit pushed permanently negative receives no gradient, so it can never recover. It is dead for the rest of training.
Leaky ReLU leaves a small slope on the negative side so there is always something to learn from.
ReLU is also not smooth at zero, and the subgradient used there is an arbitrary choice that in practice costs nothing.
Softmax takes a whole vector and returns a distribution over it, so it couples every unit to every other one.
The three above act on one number at a time, which is why they can be applied to a million-element tensor in one pass with no reduction.
Putting softmax in a hidden layer is almost always a mistake. It forces the layer's outputs to sum to one, which is a constraint nothing there asked for.
One question on this card. Every option explains itself once you pick, right or wrong.
25 concepts in this module, then a graded test with every answer explained.