Hyperparameter Tuning in machine learning

Module 01 · Machine Learning · 55 concepts

In depth / 15 lines

Hyperparameter Tuning, in more detail

Tuning is a search, and it helps to treat it like one.

Grid search tries every combination you listed.

It is thorough, and it becomes impossible as the list of settings grows.

Ten settings with five values each is nearly ten million combinations.

Random search picks combinations at random instead.

That sounds worse and usually is not, which surprises people.

Most settings barely matter, and random search wastes less time on them.

Bayesian methods use what they have already learned to choose the next attempt.

They help most when a single training run is slow and expensive.

Whatever you use, score on validation data and never on the test set.

With small data, score each attempt with cross-validation rather than one split.

Try enough combinations and eventually one wins by luck alone.

Keeping the test set sealed is how you find that out.

Start with a wide, coarse search and then narrow around whatever looked good.

And remember that better features usually beat better settings.

Want the full technical article? Wikipedia ↗

Deep dive

What is underneath Hyperparameter Tuning

The search is over a space you cannot differentiate

Parameters are learned from the data. Hyperparameters are set before training and cannot be learned by the same gradient, because the loss is not differentiable with respect to them.

\boldsymbol{\lambda}^{*} = \arg\min_{\boldsymbol{\lambda} \in \Lambda} \; \mathcal{L}_{\mathrm{val}}\big(f_{\boldsymbol{\lambda}}\big)

$\boldsymbol{\lambda}$ : one setting of every hyperparameter.
$\Lambda$ : the space of settings being searched.
$\mathcal{L}_{\mathrm{val}}$ : the validation loss after training with that setting.

Every evaluation of that objective costs a full training run, which is what makes this a search problem rather than an optimisation.

Why random search beats grid search

|\Lambda| = \prod_{i=1}^{d} n_i

A grid over $d$ hyperparameters with $n$ values each has $n^d$ points, so five values across six hyperparameters is 15,625 training runs.

P(\text{miss the top } p) = (1 - p)^{N}

Random search instead draws $N$ settings at random, and the probability of missing the best $p$ fraction of the space falls geometrically.

With $p = 0.05$ and $N = 60$ trials, the chance of landing in the top 5 percent is about 95 percent, whatever the number of hyperparameters is.

The deeper reason is that most hyperparameters barely matter. A grid spends $n^{d-1}$ runs re-testing the same value of the one that does, and random search never repeats a value.

The order to search in

Learning rate matters more than everything else combined, and it is searched on a logarithmic scale because 0.001 and 0.003 differ more than 0.1 and 0.103.

\eta \sim 10^{\,U(-5,\,-1)}

Batch size, weight decay and dropout come next. Architecture details usually come last, and most defaults are defaults because they are fine.

One tuning run, in order

Fix the search space, with log scales on anything spanning orders of magnitude.
Draw a setting at random, or from the surrogate model if using Bayesian search.
Train and score it on the validation folds.
Record the setting and its score, and stop early if the run is clearly losing.
Repeat for the budget, then retrain the winner on train and validation together.

What Bayesian optimisation adds, and when it does not

a(\boldsymbol{\lambda}) = \mathbb{E}\big[\max(0, \, \mathcal{L}^{*} - \mathcal{L}(\boldsymbol{\lambda}))\big]

It fits a cheap model of the score surface from the trials so far, then picks the next setting by expected improvement.

That spends trials where the surface looks promising or where it has never been measured.

It wins when each training run is expensive and the budget is small.

It loses when runs are cheap enough to parallelise, because random search parallelises perfectly and Bayesian search is sequential by design.

Successive halving takes the third route: start many settings, train them briefly, and keep only the best fraction each round, which is what Hyperband automates.

Test yourself

One question on this card. Every option explains itself once you pick, right or wrong.

Question 1

Why does random search often beat grid search for the same budget?

Machine Learning

55 concepts in this module, then a graded test with every answer explained.

Study this module →