Intersection over Union in computer vision

Module 03 · Computer Vision · 22 concepts

In depth / 15 lines

Intersection over Union, in more detail

A predicted box is rarely exactly the true box, so correctness needs a measure of closeness.

Intersection over Union is that measure, and it is a ratio of areas.

The intersection is the area the two boxes share.

The union is the total area they cover between them.

Divide the first by the second and you get a number between zero and one.

One means the boxes coincide exactly and zero means they do not touch.

A prediction counts as correct when it passes a threshold, and 0.5 is the usual choice.

A box at 0.49 is scored as a complete miss, which is a hard edge on a smooth quantity.

That is why detection scores are usually averaged over several thresholds.

The COCO benchmark averages from 0.5 to 0.95 in steps, which rewards tight boxes.

IoU is also how non-maximum suppression decides which duplicate boxes to remove.

Boxes overlapping the top candidate above the threshold are dropped as duplicates.

It is scale-independent, so a small object and a large one are judged the same way.

It is harsh on small objects, since a few pixels of error move the ratio a long way.

The same idea applies pixel by pixel in segmentation, where it is often called the Jaccard index.

Want the full technical article? Wikipedia ↗

Deep dive

What is underneath Intersection over Union

One number for how well two boxes agree

\mathrm{IoU} = \frac{|A \cap B|}{|A \cup B|}

$A$ : the predicted box.
$B$ : the ground truth box.

The overlap divided by the total area covered. It is 1 for a perfect match, 0 for boxes that do not touch, and scale free.

|A \cup B| = |A| + |B| - |A \cap B|

The union is computed by inclusion and exclusion rather than directly, which is why the intersection is the only geometry that has to be worked out.

Computing the intersection

x_1 = \max(x_1^A, x_1^B), \qquad y_1 = \max(y_1^A, y_1^B)

x_2 = \min(x_2^A, x_2^B), \qquad y_2 = \min(y_2^A, y_2^B)

|A \cap B| = \max(0, \; x_2 - x_1) \cdot \max(0, \; y_2 - y_1)

The clamp at zero is the whole trick. Without it, two disjoint boxes produce a negative width times a negative height, which is a positive area and a silently wrong answer.

One evaluation, in order

Compute IoU between every prediction and every ground truth box.
Match each prediction to the ground truth it overlaps most.
Call it correct if that IoU passes the threshold, usually 0.5.
Count unmatched predictions as false positives and unmatched truths as false negatives.
Sweep the confidence threshold and average the precision across it.

\mathrm{mAP@[.5:.95]}

The standard detection metric averages that whole procedure across IoU thresholds from 0.5 to 0.95, so a model cannot win by being roughly right everywhere.

Why it is a poor loss, and what replaced it

\mathcal{L} = 1 - \mathrm{IoU}

Used directly as a loss, IoU has zero gradient whenever the boxes do not overlap, which is exactly the case where the model most needs to be told which way to move.

\mathcal{L}_{\mathrm{GIoU}} = 1 - \mathrm{IoU} + \frac{|C \setminus (A \cup B)|}{|C|}

$C$ : the smallest box enclosing both.

Generalised IoU adds a penalty for the empty space in that enclosing box, so two separated boxes still produce a gradient pulling them together.

\mathcal{L}_{\mathrm{DIoU}} = 1 - \mathrm{IoU} + \frac{\rho^2(\mathbf{c}^A, \mathbf{c}^B)}{d^2}

Distance IoU penalises the distance between centres normalised by the diagonal of the enclosing box, which converges faster because it moves the box directly rather than through its area.

Where the threshold hides a decision

At 0.5 a box can be half wrong and still count as a detection, which flatters every model on large objects.

Small objects are the opposite case. A box a few pixels off on a 20 pixel object drops below the threshold, so the same absolute error is fatal at small scale and free at large scale.

IoU is also used inside non-maximum suppression to decide which overlapping predictions are duplicates, so the same number is doing two different jobs in one pipeline.

Test yourself

One question on this card. Every option explains itself once you pick, right or wrong.

Question 1

How is Intersection over Union calculated for two boxes?

Computer Vision

22 concepts in this module, then a graded test with every answer explained.

Study this module →