Perceptron in deep learning, explained in 5 lines

Module 02 · Deep Learning · 25 concepts

In depth / 15 lines

Perceptron, in more detail

The perceptron was built in 1958 to see whether a machine could learn to recognise shapes.

It takes a handful of numbers in and produces a single yes or no.

Each input is multiplied by its own weight, and the results are added together.

A bias is added to that sum, which shifts where the decision falls.

If the total clears zero the neuron fires, and otherwise it stays quiet.

Learning means nudging the weights every time the answer comes out wrong.

An input that pushed towards the wrong answer has its weight reduced a little.

Repeat that over the data and the weights settle into something that works.

What it draws is a straight line, or a flat plane once you have more than two inputs.

Anything the line can separate, the perceptron can learn, and nothing else.

Exclusive-or is the famous counterexample, since no single straight line separates it.

That result stalled the whole field for over a decade.

The escape was to stack perceptrons in layers, so later ones see what earlier ones found.

That stack is a neural network, and the perceptron is the unit it is built from.

Every neuron in a modern model is still doing this same weighted sum.

Want the full technical article? Wikipedia ↗

Deep dive

What is underneath Perceptron

One neuron, written out

y = \begin{cases} 1 & \text{if } \mathbf{w}^\top \mathbf{x} + b > 0 \ 0 & \text{otherwise} \end{cases}

$\mathbf{x}$ : the input features.
$\mathbf{w}$ : one weight per feature.
$b$ : the bias, which shifts the threshold away from zero.

The weighted sum is a number. The step function turns it into a decision.

Geometrically $\mathbf{w}^\top \mathbf{x} + b = 0$ is a hyperplane, $\mathbf{w}$ is the direction perpendicular to it, and the bias is how far it sits from the origin.

The update rule

\mathbf{w} \leftarrow \mathbf{w} + \eta \, (y - \hat{y}) \, \mathbf{x}

$y$ : the true label.
$\hat{y}$ : what the perceptron answered.
$\eta$ : the learning rate.

When the answer is right, $y - \hat{y}$ is zero and nothing moves. Learning only happens on mistakes.

When a positive example is called negative, $y - \hat{y}$ is $+1$ and the input is added to the weights, tilting the plane towards that point.

When a negative is called positive it is subtracted instead. The whole algorithm is those two cases.

One pass, in order

Take a training example.
Compute the weighted sum and threshold it.
Compare with the true label.
On a mistake, add or subtract the input from the weights.
Move to the next example, and repeat the sweep until no mistake is made.

The guarantee, and its price

k \leq \left( \frac{R}{\gamma} \right)^{2}

$R$ : the radius of the smallest ball containing the data.
$\gamma$ : the margin, the distance from the boundary to the nearest point.

If the data is linearly separable, the perceptron makes at most that many mistakes before it stops making any. That is Rosenblatt's convergence theorem from 1958.

If the data is not separable, it never converges. It cycles forever, and it has no notion of a best approximate answer.

Why one is not enough

\mathrm{XOR}(0,0) = 0, \quad \mathrm{XOR}(0,1) = 1, \quad \mathrm{XOR}(1,0) = 1, \quad \mathrm{XOR}(1,1) = 0

No single straight line separates those four points, so no setting of two weights and a bias computes XOR.

Minsky and Papert proved that in 1969 and the field's funding collapsed for a decade behind it.

The fix was already implied: stack a layer of these and the first layer bends the space so the second can cut it with a plane.

\sigma(z) = \frac{1}{1 + e^{-z}}

What stacking needed was a differentiable threshold, because a step function has a gradient of zero everywhere and cannot be trained by backpropagation.

That substitution is the difference between a perceptron and a neural network.

Test yourself

One question on this card. Every option explains itself once you pick, right or wrong.

Question 1

What could a single perceptron never do?

Deep Learning

25 concepts in this module, then a graded test with every answer explained.

Study this module →