Implementing Architectures from Scratch

Building an architecture yourself exposes everything reading it lets you skip.

You cannot hand-wave a tensor shape into existence.

Write the minimal version first, with no optimisations and no configuration.

Hard-code the sizes, since generality is what makes a first implementation unreadable.

Test each component in isolation with tiny tensors whose shapes you can check by hand.

Attention on a sequence of four tokens is verifiable on paper.

Print shapes at every step, since most bugs are a transposed or broadcast dimension.

Check against a reference implementation with the same weights and input.

Matching outputs to a few decimal places is the only real confirmation.

Overfit a single batch deliberately, since a correct model can memorise ten examples.

If it cannot, the bug is in the model rather than in the training setup.

Attention, a small transformer and a CNN are the classic exercises for good reason.

Each takes an afternoon and teaches more than a week of reading.

Then read a production implementation and see what the optimisations cost in clarity.

Keep the toy version, since it stays readable in a way the fast one never will.