Linear Regression in machine learning

Module 01 · Machine Learning · 55 concepts

In depth / 15 lines

Linear Regression, in more detail

Linear regression assumes each feature pushes the answer up or down by a fixed amount.

Double the area and its contribution to the price doubles too.

That assumption is often wrong and still often useful.

The great advantage is that you can read what it learned.

Each weight says how much that feature moves the prediction.

That matters wherever you have to explain a decision to someone.

It is also fast, and it is a proper baseline to beat.

If a complicated model cannot beat it, that is a result worth knowing.

It is easily thrown off by extreme values, because it minimises squared error.

One wildly priced house can tilt the whole line.

Features that move together make the individual weights unstable and hard to trust.

The predictions can still be fine while the explanation is nonsense.

You can capture curves by adding squared or log versions of features.

At that point you are doing feature engineering, not changing the model.

Always look at the errors, because a pattern in them means something is missing.

Want the full technical article? Wikipedia ↗

Deep dive

What is underneath Linear Regression

The model

\hat{y} = w_1 x_1 + w_2 x_2 + \cdots + w_d x_d + b = \mathbf{w}^{\top} \mathbf{x} + b

$\mathbf{x}$ : the feature vector for one example.
$\mathbf{w}$ : one weight per feature.
$b$ : the intercept, the prediction when every feature is zero.

Each weight is the change in the prediction per unit change in that feature, holding the others fixed.

That reading is the reason linear models are still used wherever an explanation is required.

What it is fitting

\mathcal{L}(\mathbf{w}) = \sum_{i=1}^{n} \left( y_i - \mathbf{w}^{\top}\mathbf{x}_i \right)^2

Sum of squared residuals, which is the ordinary least squares objective.

Squaring is not arbitrary: it makes the objective differentiable everywhere, and it makes the fitted line the maximum-likelihood estimate under Gaussian noise.

The closed form

Unlike almost everything else in this course, this one has an exact answer that needs no iteration.

\hat{\mathbf{w}} = \left( X^{\top} X \right)^{-1} X^{\top} \mathbf{y}

$X$ : the design matrix, one row per example, one column per feature.

Set the derivative of the loss to zero, rearrange, and that falls out. It is called the normal equation.

\frac{\partial \mathcal{L}}{\partial \mathbf{w}} = -2X^{\top}\left( \mathbf{y} - X\mathbf{w} \right) = 0

Why nobody uses the closed form at scale

O(d^3) \text{ to invert, } O(nd^2) \text{ to form } X^{\top}X

Inverting a $d \times d$ matrix is cubic in the number of features, so at ten thousand features it is already slower than gradient descent.

Worse, $X^{\top}X$ is singular whenever two features are perfectly correlated, or whenever there are more features than examples, and then the inverse does not exist at all.

Adding an L2 penalty fixes both: it makes the matrix invertible by construction.

\hat{\mathbf{w}}_{\text{ridge}} = \left( X^{\top} X + \lambda I \right)^{-1} X^{\top} \mathbf{y}

One fit, in order

Stack the examples into $X$ and the targets into $\mathbf{y}$ .
Add a column of ones so the intercept is just another weight.
Either solve the normal equation, or run gradient descent on the squared loss.
Read the weights as per-unit effects, if the features are on comparable scales.
Check the residuals: if they show a pattern, the relationship was not linear.

Test yourself

One question on this card. Every option explains itself once you pick, right or wrong.

Question 1

When does linear regression struggle most?

Machine Learning

55 concepts in this module, then a graded test with every answer explained.

Study this module →