Feature Scaling in machine learning

Module 01 · Machine Learning · 55 concepts

In depth / 15 lines

Feature Scaling, in more detail

Columns in real data arrive on wildly different scales.

Age runs from 0 to 100 while income runs from 0 to 200,000.

Any algorithm that measures distance will be dominated by income and effectively ignore age.

A thousand pounds of income is not a thousand times more important than a year of age.

Scaling puts every column onto comparable footing so the algorithm can weigh them fairly.

Standardisation subtracts the mean and divides by the standard deviation.

The result is centred on zero with a spread of one, and outliers survive rather than squash everything.

Min-max scaling maps the smallest value to zero and the largest to one.

That is neat, and a single extreme value crushes everything else into a narrow band.

KNN, SVM, K-Means and PCA all depend on distance and therefore all need scaling.

Neural networks need it too, since unscaled inputs make gradients uneven and training slow.

Trees, forests and boosted trees split one column at a time and ignore scale entirely.

Fit the scaler on the training split and apply those same numbers to test data.

Fitting on everything leaks the test distribution into training, which quietly inflates your score.

It is one line of code and one of the most common reasons a model underperforms.

Want the full technical article? Wikipedia ↗

Deep dive

What is underneath Feature Scaling

The two transforms

x' = \frac{x - \mu}{\sigma} \qquad x' = \frac{x - x_{\min}}{x_{\max} - x_{\min}}

The first is standardisation: subtract the mean, divide by the standard deviation. The result has mean 0 and standard deviation 1, and is unbounded.

The second is min-max normalisation. The result is squeezed into the interval from 0 to 1, and it is bounded but sensitive to a single outlier.

$\mu, \sigma$ : the mean and standard deviation of that feature, over the training set only.

Why it changes the answer at all

Gradient descent takes the same step size in every direction. If one feature ranges over 0 to 1 and another over 0 to 100,000, the loss surface is a long thin valley.

\Delta w_j \propto \frac{\partial \mathcal{L}}{\partial w_j} \propto x_j

The gradient for each weight is proportional to its feature's magnitude, so the large-scale feature produces enormous updates and the small-scale one produces almost none.

A learning rate small enough to be stable for the first is far too small for the second, and training crawls along the valley floor instead of crossing it.

Scaling makes the valley round, and a round valley can be descended in a few steps.

Which models care and which do not

Distance-based methods care completely: k-nearest neighbours, k-means and SVMs all compute distances, and an unscaled feature dominates every distance.

d(\mathbf{a},\mathbf{b}) = \sqrt{\sum_j (a_j - b_j)^2}

Regularised models care, because a penalty on the weights is a penalty measured in the units the features happen to use.

Tree-based models do not care at all. A decision tree splits on thresholds within one feature and never compares two features' magnitudes, so any monotonic rescaling gives an identical tree.

The mistake that leaks the test set

Compute the mean and standard deviation on the training set only.
Store those two numbers as part of the model.
Apply them to the validation set.
Apply the same ones to the test set, and to every request in production.
Never recompute them on data the model has not been trained on.

Fitting the scaler on the whole dataset before splitting lets information about the test set reach the model, and the result is a score that does not survive deployment.

Test yourself

One question on this card. Every option explains itself once you pick, right or wrong.

Question 1

Which of these does not care whether the features are scaled?

Machine Learning

55 concepts in this module, then a graded test with every answer explained.

Study this module →