Mixture-of-Experts in LLMs, explained in 5 lines

Module 16 · Advanced AI · 23 concepts

In depth / 15 lines

Mixture-of-Experts, in more detail

A mixture-of-experts model contains many expert subnetworks instead of one.

Each expert is usually a feed-forward block, and there might be eight or a hundred and twenty-eight.

A small router decides which experts should handle each token.

Only a couple are used, so most of the model is idle for any given token.

That is the entire trick: enormous total parameters and small active parameters.

A model with 400 billion total might use 20 billion per token.

You get the capability of a large model at the compute of a much smaller one.

Different experts specialise, though not in ways that map neatly onto topics.

The router is trained alongside everything else.

Load balancing is the practical difficulty, since a router can route everything to one favourite.

An auxiliary loss encourages even use across experts.

Memory is the real cost, since every expert must be held even if rarely used.

That makes them cheap to compute and expensive to host.

They also complicate distributed serving, since experts are spread across devices.

Mixtral, DeepSeek and several frontier models use this design.

Want the full technical article? Wikipedia ↗

Deep dive

What is underneath Mixture-of-Experts

Parameters you have without paying for them

\mathbf{y} = \sum_{i \in \mathcal{T}} g_i(\mathbf{x}) \, E_i(\mathbf{x})

$E_i$ : one expert, usually a feed-forward block.
$g_i$ : the gate's weight for that expert.
$\mathcal{T}$ : the selected top- $k$ experts.

A dense layer uses every parameter for every token. A mixture layer has many experts and uses a few per token.

g(\mathbf{x}) = \mathrm{softmax}\big( \mathrm{top\text{-}}k(W_g \mathbf{x}) \big)

The router is one small matrix producing a score per expert, from which the highest $k$ are kept and renormalised.

8 \text{ experts}, \; k = 2 \implies \text{4 times the parameters, same FLOPs}

That ratio is the entire proposition: capacity scales with the number of experts, and compute scales with $k$ .

Load balancing, or it collapses

\mathcal{L}_{\mathrm{aux}} = \alpha \, N \sum_{i=1}^{N} f_i \, P_i

$f_i$ : the fraction of tokens routed to expert $i$ .
$P_i$ : the mean gate probability for it.

Left alone, the router sends almost everything to a few experts. Those get more training, become better, and attract more traffic still.

The auxiliary loss is minimised when the load is uniform, and it is added to the training objective with a small coefficient.

\text{capacity factor} \approx 1.25

Each expert also has a fixed buffer per batch.

Tokens beyond it are dropped and pass through the residual connection unprocessed, which bounds the memory at the cost of occasionally skipping a layer.

One token, in order

The token's hidden state reaches the mixture layer.
The router scores every expert and keeps the top $k$ .
The token is dispatched to those experts, across devices if they are sharded.
Each expert computes its output.
The outputs are combined by the gate weights and added back into the residual stream.

Steps three and five are a network operation in a distributed setup, and they are usually the bottleneck rather than the arithmetic.

What it costs that dense models do not

\text{memory} \propto \text{total parameters}

Every expert must be resident even though only a fraction runs, so a mixture model needs the memory of its full size while using the compute of its active size.

That is the honest trade: it is a way of spending memory and bandwidth to buy quality at fixed compute, not a free lunch.

Training is also less stable. The router is discrete, the gradient reaches only the selected experts, and small changes in routing early on can decide which experts ever become useful.

Why it is now standard

\text{active} \approx 15 \text{ to } 25\% \text{ of total}

Most current frontier models are mixtures, because inference cost is set by the active parameters and quality tracks the total.

Fine-tuning them is harder, since a small dataset routes to a subset of experts and leaves the rest untouched, which is a real problem for anyone adapting one rather than training it.

Test yourself

One question on this card. Every option explains itself once you pick, right or wrong.

Question 1

In a Mixture-of-Experts model, why is compute lower than total parameter count suggests?

Advanced AI

23 concepts in this module, then a graded test with every answer explained.

Study this module →