Adversarial Attacks in AI security

Module 15 · AI Security · 16 concepts

In depth / 15 lines

Adversarial Attacks, in more detail

Adversarial attacks craft inputs that fool a model while appearing normal to a person.

The classic vision example changes pixels imperceptibly and flips the classification.

A stop sign with a few small stickers can be read as a speed limit sign.

The changes are calculated, not random, and follow the model's own gradients.

They exist because decision boundaries are strange in high-dimensional space.

A point can sit close to a boundary in a direction no human would notice.

In text the equivalent is small rewordings that bypass a content filter.

Character substitutions, spacing tricks and synonyms all shift the model's reading.

Attacks transfer between models more often than intuition suggests.

So an attacker can develop against an open model and use it against a closed one.

That undermines the assumption that a private model is a safe one.

Adversarial training adds such examples to the training set and helps.

It does not eliminate the problem, and it usually costs a little ordinary accuracy.

Input sanitisation, ensembles and detection each raise the cost of an attack.

For anything safety-critical, do not let a single model decision be the last word.

Want the full technical article? Wikipedia ↗

Deep dive

What is underneath Adversarial Attacks

A perturbation too small to see

\mathbf{x}_{\mathrm{adv}} = \mathbf{x} + \boldsymbol{\delta}, \qquad \|\boldsymbol{\delta}\|_{\infty} \leq \epsilon

$\epsilon$ : the budget, often 8 out of 255 for images, which is invisible on a screen.

The constraint is what makes this an attack rather than a different input: the image still looks identical and the label still obviously applies.

\max_{\boldsymbol{\delta}} \; \mathcal{L}\big(f(\mathbf{x} + \boldsymbol{\delta}), \; y\big) \quad \text{subject to} \quad \|\boldsymbol{\delta}\|_{\infty} \leq \epsilon

Training minimises the loss with respect to the weights. An attack maximises the same loss with respect to the input, using the same gradient.

FGSM, one step

\boldsymbol{\delta} = \epsilon \cdot \mathrm{sign}\big( \nabla_{\mathbf{x}} \mathcal{L}(f(\mathbf{x}), y) \big)

Take the gradient of the loss with respect to the pixels, keep only its sign, and move every pixel to the edge of the budget.

The sign is used rather than the gradient itself because the constraint is on the maximum change per pixel, so the largest allowed step is the corner of the box.

It takes one backward pass, it is the cheapest attack there is, and it succeeds on an undefended model most of the time.

PGD, which is the real benchmark

\mathbf{x}^{t+1} = \Pi_{\mathcal{B}(\mathbf{x}, \epsilon)}\Big( \mathbf{x}^{t} + \alpha \, \mathrm{sign}\big(\nabla_{\mathbf{x}} \mathcal{L}\big) \Big)

$\Pi$ : projection back into the allowed ball after each step.
$\alpha$ : the step size, smaller than $\epsilon$ .

Projected gradient descent takes many small steps instead of one large one, re-projecting each time, and it starts from a random point inside the ball.

It usually runs for 20 to 100 steps.

It is far stronger than FGSM, and a defence that only reports FGSM accuracy is reporting almost nothing.

One evaluation, in order

Fix the threat model: the norm, the budget, and whether the attacker sees the weights.
Run PGD with several random restarts, not a single one.
Report accuracy under attack, not the attack's success rate on a subset.
Check that the gradients are not simply unusable, which flatters a defence without adding robustness.
Try a transfer attack from a substitute model, since a real attacker often has no access to the weights.

Why it happens at all

f(\mathbf{x}) \approx f(\mathbf{x}_0) + \nabla f^{\top}(\mathbf{x} - \mathbf{x}_0)

Networks are locally close to linear, and in a space of 150,000 dimensions a small change per pixel adds up to a large change in the logits.

\boldsymbol{\delta}^{\top} \mathbf{w} = \epsilon \sum_i |w_i|

That sum grows with dimension, which is why high resolution inputs are more vulnerable rather than less.

Adversarial examples also transfer between models trained on the same data, which means they exploit something about the data and the class of functions rather than a quirk of one network.

The only defence that has held

\min_{\theta} \; \mathbb{E}\Big[ \max_{\|\boldsymbol{\delta}\| \leq \epsilon} \mathcal{L}\big(f_{\theta}(\mathbf{x} + \boldsymbol{\delta}), y\big) \Big]

Adversarial training generates attacks during training and learns on them, which is a minimax problem solved by running PGD inside the training loop.

It costs several times the training compute, and it reduces clean accuracy by a few points, and after a decade of proposals it remains the only method that has not been broken.

Certified defences give a provable radius instead of an empirical one, and the radii they can prove are still far smaller than the perturbations attackers use.

Test yourself

One question on this card. Every option explains itself once you pick, right or wrong.

Question 1

What do adversarial attacks exploit?

AI Security

16 concepts in this module, then a graded test with every answer explained.

Study this module →