Scaling Laws in LLMs, explained in 5 lines

Module 06 · LLMs · 22 concepts

In depth / 15 lines

Scaling Laws, in more detail

Scaling laws describe how a model improves as you spend more on it.

The three inputs are parameters, training data and compute.

Plot loss against any of them on log axes and you get a straight line.

That regularity holds across many orders of magnitude, which is genuinely surprising.

It means a run costing millions can be predicted from a series of cheap small ones.

That predictability is what made the spending defensible to the people writing the cheques.

The three have to grow together, since starving one wastes the others.

The original guidance leaned heavily towards more parameters.

The Chinchilla work in 2022 rebalanced it, showing models were badly undertrained.

Roughly twenty tokens of training data per parameter turned out to be far better.

A smaller model trained on more data beat a larger one at the same total cost.

It is also cheaper to run afterwards, which compounds the win.

The curves describe loss, not usefulness, and those are not the same thing.

Improvements on the loss curve show up unevenly across actual tasks.

The laws also assume plentiful fresh data, which is the constraint now starting to bind.

Want the full technical article? Wikipedia ↗

Deep dive

What is underneath Scaling Laws

Loss falls as a power law

Across many orders of magnitude, a language model's loss follows a straight line on a log-log plot against parameters, data or compute.

L(N) = \left( \frac{N_c}{N} \right)^{\alpha_N}

$N$ : number of parameters.
$N_c, \alpha_N$ : fitted constants; $\alpha_N$ is around 0.076 in the original work.

A power law means constant proportional returns: every tenfold increase in size buys the same absolute drop in loss as the previous tenfold did.

That is both the good news and the bad. It never stops working, and it never gets cheaper.

The joint form

L(N, D) = \left( \frac{N_c}{N} \right)^{\alpha_N} + \left( \frac{D_c}{D} \right)^{\alpha_D} + L_\infty

$D$ : number of training tokens.
$L_\infty$ : the irreducible loss, the entropy of the text itself.

The two terms add, so whichever is larger dominates. A huge model on too little data is bounded by the second term, and no amount of extra parameters touches it.

The compute-optimal ratio

Given a fixed compute budget, there is one best way to split it between making the model bigger and training it longer.

C \approx 6 N D

Minimising the joint loss under that constraint gives the Chinchilla result.

N_{\text{opt}} \propto C^{0.5} \qquad D_{\text{opt}} \propto C^{0.5}

\frac{D}{N} \approx 20 \text{ tokens per parameter}

Both should grow at the same rate, in equal proportion.

Why that finding mattered

GPT-3 had 175 billion parameters and 300 billion training tokens, a ratio of about 1.7.

By the Chinchilla ratio it should have had 3.5 trillion tokens, or been about eight times smaller for the same compute.

Chinchilla itself had 70 billion parameters and 1.4 trillion tokens.

A 70B model trained correctly beat a 175B model trained the old way, using the same compute. That is the whole result, and it redirected the field from parameter count to data.

What the laws do not say

They predict loss, not capability. A given loss does not tell you whether the model can do arithmetic.
They are fitted to a fixed architecture and data distribution, and change when either does.
They say nothing about inference cost, which is why models are now trained well past the compute-optimal point on purpose.
A model that will serve billions of requests should be smaller and trained longer than Chinchilla says, because inference is paid forever.
$L_\infty$ is a real floor, and how far away it is nobody knows.

Test yourself

One question on this card. Every option explains itself once you pick, right or wrong.

Question 1

What did the Chinchilla result change about how large models are trained?

LLMs

22 concepts in this module, then a graded test with every answer explained.

Study this module →