Neural Networks

A neural network is a stack of layers, and each layer is a row of neurons.

The input layer is your data, one number per feature.

Hidden layers sit in between and are where the actual work happens.

The output layer produces whatever you asked for, a number or a set of class scores.

Every neuron takes a weighted sum of the layer before it and applies an activation.

The activation is what stops the whole stack collapsing into one straight line.

Early layers tend to find simple things, like an edge in an image.

Later layers combine those into bigger things, like a shape and then a face.

Nobody assigns those roles, since they fall out of training on their own.

Width is how many neurons sit in a layer, and depth is how many layers there are.

Depth is usually the cheaper way to buy capability, which is where the word deep comes from.

A network with enough neurons can approximate almost any function you like.

That guarantee says nothing about whether training will actually find it.

More capacity also means more room to memorise the training set.

So the size of the network is a choice you tune, not a number you maximise.