Parameters in LLMs, explained in 5 lines

Module 06 · LLMs · 22 concepts

In depth / 15 lines

Parameters in LLMs, in more detail

Parameters are the learned numbers inside the model, and they are all it knows.

A 7B model has seven billion of them, which is where the name comes from.

They are spread across attention weights, feed-forward layers and the embedding table.

Most of them sit in the feed-forward layers, not in attention.

Nothing is stored as a fact you could look up, since knowledge is smeared across the whole set.

Memory needed is roughly two bytes per parameter at half precision.

So a 7B model needs about 14 GB just to hold its weights.

Quantising to 4 bits brings that near 4 GB, which is why laptops can run these at all.

Training needs several times more, for gradients and optimizer state.

More parameters generally means more capability, more cost and more latency.

The relationship is smooth and predictable rather than sudden.

Data quality and quantity matter at least as much as raw count.

A well-trained 7B model beats a badly trained 70B one on most tasks.

Mixture-of-experts models quote a total count but only use a fraction per token.

Parameter count is a rough guide to cost and a poor guide to quality.

Want the full technical article? Wikipedia ↗

Deep dive

What is underneath Parameters in LLMs

Counting them from the architecture

A model's parameter count is not a mystery number from a press release. It follows from four dimensions.

N \approx 12 \, L \, d^2

$L$ : the number of transformer blocks.
$d$ : the model width, $d_{\text{model}}$ .

The 12 comes from adding up one block: four $d^2$ in attention and eight in the feed-forward layer.

Where the twelve comes from

\text{attention: } W_Q, W_K, W_V, W_O = 4 d^2

\text{FFN: } W_1 \, (d \times 4d) + W_2 \, (4d \times d) = 8 d^2

So two thirds of a transformer's weights sit in the feed-forward layers, not in the attention everyone talks about.

Normalisation adds $2d$ per sublayer and biases add $d$ or so, both negligible against $d^2$ terms.

The embeddings, which are separate

N_{\text{embed}} = V \times d

$V$ : vocabulary size, typically 32,000 to 256,000.

At $V = 128{,}000$ and $d = 4096$ that is 524 million parameters in the input embedding alone, and the same again in the output projection unless the two are tied.

For a small model the embeddings can be most of the parameters. For a large one they are a rounding error, which is why the $12Ld^2$ approximation gets better as models get bigger.

Two real models

\text{Llama-2 7B: } L = 32, \; d = 4096 \implies 12 \times 32 \times 4096^2 = 6.4\text{B}

\text{70B: } L = 80, \; d = 8192 \implies 12 \times 80 \times 8192^2 = 64.4\text{B}

Both land within about 10% of the advertised number, and the gap is the embeddings and the gated feed-forward layer's third matrix.

What the count buys you

Memory for the weights: multiply by 2 bytes for fp16.
Memory to fine-tune fully: roughly 16 bytes per parameter with Adam, so about 16x the weights.
Arithmetic per token: about $2N$ floating-point operations for a forward pass.
Arithmetic per training token: about $6N$ , since backward costs twice forward.
Total training compute: about $6ND$ for $D$ training tokens.

Those five estimates are enough to answer most practical questions about a model without looking anything up.

Test yourself

One question on this card. Every option explains itself once you pick, right or wrong.

Question 1

Roughly how much memory do the weights of a 7B model need at half precision?

LLMs

22 concepts in this module, then a graded test with every answer explained.

Study this module →