Reading Mathematical Notation in AI research

Module 17 · Research Level · 11 concepts

In depth / 15 lines

Reading Mathematical Notation, in more detail

Notation exists to compress a precise statement, not to make a paper look serious.

An equation that looks impenetrable is usually a familiar idea in unfamiliar symbols.

The first move is to name every symbol before trying to read the whole line.

Ask of each one whether it is a number, a vector, a matrix, a function or an index.

Lower case usually means a scalar or a vector, and upper case a matrix or a set.

Subscripts index into something, and superscripts are often a power or a layer number.

A capital sigma is a sum, and what sits under it says what the sum runs over.

A capital pi is a product, and it is read in exactly the same way.

An E with a subscript is an expectation, which means an average over that distribution.

The argmax of an expression is the input that makes it largest, not the largest value.

A hat usually marks a prediction, and a star usually marks an optimal or true value.

Papers define their notation once, in a short early paragraph, and never repeat it.

Reading that paragraph properly is worth more than rereading the equations later.

Writing an equation out in words, or in code, is the fastest way to check you have it.

Fluency comes from doing this slowly perhaps thirty times, after which it stops being slow.

Want the full technical article? Wikipedia ↗

Deep dive

What is underneath Reading Mathematical Notation

The symbols that carry most of a paper

\sum_{i=1}^{n} x_i, \qquad \prod_{i=1}^{n} x_i

A sum and a product over an index. The subscript says where it starts, the superscript where it ends, and the body is what is being combined.

\mathbb{E}_{x \sim p}\big[ f(x) \big] = \sum_{x} p(x) f(x)

An expectation is a weighted average, and the subscript names the distribution the sample comes from. It is the most common symbol in machine learning papers and the most often skipped.

\arg\max_{a} f(a)

Not the largest value of $f$ , which is $\max$ , but the input that produces it. The distinction matters in every decision rule ever written.

Shapes, which are half the meaning

x \in \mathbb{R}, \qquad \mathbf{x} \in \mathbb{R}^{d}, \qquad X \in \mathbb{R}^{n \times d}

Lower case italic is a scalar, bold lower case a vector, upper case a matrix. The set on the right says the size, and reading it is how you check an equation without understanding it.

W \in \mathbb{R}^{m \times n}, \; \mathbf{x} \in \mathbb{R}^{n} \implies W\mathbf{x} \in \mathbb{R}^{m}

The inner dimensions must agree and they vanish. That single rule resolves most confusion about which way round a matrix multiply goes.

\mathbf{a}^{\top}\mathbf{b} \in \mathbb{R}, \qquad \mathbf{a}\mathbf{b}^{\top} \in \mathbb{R}^{d \times d}

Distance, size and probability

\|\mathbf{x}\|_2 = \sqrt{\sum_i x_i^{2}}, \qquad \|\mathbf{x}\|_1 = \sum_i |x_i|

The subscript on a norm says which one. Two is the ordinary length, one is the sum of absolute values, and infinity is the largest single element.

p(y \mid x)

The bar means given. Everything to its right is assumed known, and this is the object almost every supervised model estimates.

x \sim \mathcal{N}(\mu, \sigma^{2})

The tilde means drawn from, and the parameters in the brackets describe the distribution rather than the sample.

Reading a loss function, in order

Find the argmin or argmax, which says what is being chosen.
Read the subscript under it, which says which quantities are free to vary.
Identify the expectation and what it averages over.
Split the expression at the plus signs, since a loss is almost always a fit term and a penalty.
Check the shapes, and only then work out why each term is there.

Conventions worth knowing before they confuse you

\hat{y}

A hat means an estimate of the thing underneath it. A bar means an average, a star means optimal, and a tilde often means a corrupted or approximate version.

\theta, \; \phi, \; \psi

Greek letters are parameters by convention, and Latin letters are data. When a paper writes $f_{\theta}(x)$ it is separating what is learned from what is given.

\nabla_{\theta} \mathcal{L}

The gradient symbol with a subscript means differentiate with respect to that variable and nothing else, which is how an optimisation objective states what it will move.

\propto, \; \approx, \; \triangleq

Proportional to hides a constant that does not matter. Approximately equal usually hides an assumption stated in the text.

The last means defined as, and it is introducing notation rather than claiming a result.

Test yourself

One question on this card. Every option explains itself once you pick, right or wrong.

Question 1

What is the recommended way into an unfamiliar equation?

Research Level

11 concepts in this module, then a graded test with every answer explained.

Study this module →