Efficient Architectures in computer vision

Module 03 · Computer Vision · 22 concepts

In depth / 15 lines

Efficient Architectures, in more detail

Accuracy on a benchmark says nothing about whether a model can run where you need it.

A phone, a drone or a security camera has a power budget measured in single-digit watts.

Efficient architectures are designs that trade a point or two of accuracy for an order of magnitude less compute.

The central idea is the depthwise separable convolution, which factors one operation into two.

A normal convolution filters space and mixes channels in a single expensive step.

The separable version filters each channel on its own, then mixes channels with a one by one convolution.

For a typical layer that costs roughly eight or nine times fewer multiplications.

MobileNet is essentially a stack of those blocks with a width multiplier to trade size for accuracy.

Its later versions added inverted residuals, expanding channels inside the block and shrinking them at the edges.

EfficientNet attacked a different question: given more compute, what should you enlarge?

Its answer was that depth, width and input resolution must grow together by a fixed ratio.

Scaling only one of the three saturates quickly, which is what earlier hand-tuned models kept doing.

Many of these designs came from neural architecture search rather than from human intuition.

Parameter count and FLOPs are both poor proxies, because memory traffic often dominates real latency.

The only trustworthy number is milliseconds measured on the actual target device.

Deep dive

What is underneath Efficient Architectures

Where the arithmetic actually goes

\mathrm{FLOPs} = 2 H W K^2 C_{\mathrm{in}} C_{\mathrm{out}}

A standard convolution multiplies four things that are all large at once, and the product is what makes a network too slow for a phone.

2 \cdot 14^2 \cdot 3^2 \cdot 512 \cdot 512 \approx 9.2 \times 10^{8}

One such layer is nearly a billion operations per image, and a network has fifty of them.

The efficient architectures all attack the same term: they refuse to mix space and channels in the same operation.

Depthwise separable convolution

\text{depthwise} : K^2 C_{\mathrm{in}} \text{ parameters, one filter per channel}

\text{pointwise} : 1 \times 1 \times C_{\mathrm{in}} \times C_{\mathrm{out}} \text{ parameters}

The first step filters each channel on its own and mixes nothing. The second mixes channels and looks at no neighbours.

\frac{K^2 C_{\mathrm{in}} + C_{\mathrm{in}} C_{\mathrm{out}}}{K^2 C_{\mathrm{in}} C_{\mathrm{out}}} = \frac{1}{C_{\mathrm{out}}} + \frac{1}{K^2}

For a 3 by 3 kernel and any reasonable channel count that ratio is about $1/9$ , so the same receptive field and the same channel mixing cost a ninth as much.

MobileNet is that substitution applied everywhere, and it is the single largest efficiency win in the field.

The inverted residual

C \xrightarrow{1 \times 1} tC \xrightarrow{3 \times 3 \text{ dw}} tC \xrightarrow{1 \times 1} C

$t$ : the expansion factor, usually 6.

A classical bottleneck narrows, convolves, and widens. An inverted residual widens, filters cheaply in the wide space, and narrows again.

It is the right way round once the middle convolution is depthwise.

A depthwise convolution costs almost nothing per channel, and the expensive part is the pointwise mixing at the ends.

The residual connection joins the thin tensors rather than the fat ones, which is what keeps the memory footprint small on a device with no room to spare.

Compound scaling

\text{depth} = \alpha^{\phi}, \quad \text{width} = \beta^{\phi}, \quad \text{resolution} = \gamma^{\phi}

\alpha \beta^2 \gamma^2 \approx 2

Scaling only one of depth, width or resolution saturates quickly. EfficientNet's result is that all three must grow together, in a fixed ratio, for the extra compute to pay.

The constraint says doubling the compute means multiplying the three by factors whose product of powers is 2, since width and resolution both enter the cost quadratically.

One efficient design, in order

Replace every 3 by 3 convolution with a depthwise pair.
Put the residual connections between the narrow tensors.
Use the smallest resolution the task tolerates, since cost is quadratic in it.
Scale depth, width and resolution together rather than one at a time.
Measure latency on the target device, not FLOPs.

Why FLOPs are the wrong target

\text{arithmetic intensity} = \frac{\mathrm{FLOPs}}{\text{bytes moved}}

A depthwise convolution does very little arithmetic per byte it reads, so it is limited by memory bandwidth and runs far below the hardware's peak.

That is why a model with a quarter of the FLOPs is often only twice as fast, and why two architectures with identical FLOPs can differ threefold in latency.

Anything claiming efficiency without a measured latency on real hardware is reporting a proxy, and the proxy is known to be loose.

Test yourself

One question on this card. Every option explains itself once you pick, right or wrong.

Question 1

What makes a depthwise separable convolution cheaper than a standard one?

Computer Vision

22 concepts in this module, then a graded test with every answer explained.

Study this module →