Logistic Regression in machine learning

Module 01 · Machine Learning · 55 concepts

In depth / 15 lines

Logistic Regression, in more detail

Logistic regression predicts a probability, not a class.

It does the same weighted sum as linear regression.

Then it squashes the result into the range between zero and one.

That squashing is what makes the output usable as a probability.

Turning it into a decision is a separate step, and that is the threshold.

Because the steps are separate, you can move the threshold whenever you like.

That is genuinely useful when the two kinds of mistake cost different amounts.

Like linear regression, you can read the weights and learn something real.

It remains the standard first attempt for two-class problems.

It cannot draw a curved boundary unless you hand it curved features.

It assumes your classes can be separated by a straight line somewhere.

Perfectly separable data actually breaks it, by pushing the weights to infinity.

Regularization prevents that, which is why it is usually switched on by default.

Its probabilities are reasonably trustworthy, which is not true of every model.

That makes it a good choice when you need the number and not just the label.

Want the full technical article? Wikipedia ↗

Deep dive

What is underneath Logistic Regression

It is linear, then squashed

Linear regression can output 4.7 or minus 12, and neither is a probability. Logistic regression puts the linear score through a function that lands in the unit interval.

z = \mathbf{w}^{\top}\mathbf{x} + b \qquad \hat{y} = \sigma(z) = \frac{1}{1 + e^{-z}}

$z$ : the raw linear score, called the logit.
$\sigma$ : the logistic sigmoid.

At $z = 0$ the output is exactly 0.5. Large positive $z$ approaches 1, large negative approaches 0, and it never reaches either.

What the weights actually mean

Invert the sigmoid and the linear part reappears, applied to something specific.

\log \frac{\hat{y}}{1 - \hat{y}} = \mathbf{w}^{\top}\mathbf{x} + b

The left side is the log-odds. So logistic regression is a linear model of the log-odds, not of the probability.

\frac{\hat{y}}{1-\hat{y}} \times e^{w_j} \quad \text{per unit increase in } x_j

That means a weight of 0.7 multiplies the odds by $e^{0.7} \approx 2$ for each unit of that feature: a constant multiplicative effect, not a constant additive one.

The loss, and why not squared error

\mathcal{L} = -\frac{1}{n}\sum_{i=1}^{n} \left[ y_i \log \hat{y}_i + (1 - y_i) \log (1 - \hat{y}_i) \right]

This is binary cross-entropy. Squared error on top of a sigmoid gives a non-convex loss with flat regions where the gradient vanishes, and training stalls there.

Cross-entropy composed with the sigmoid is convex, and its gradient is famously clean.

\frac{\partial \mathcal{L}}{\partial \mathbf{w}} = \frac{1}{n} \sum_{i=1}^{n} \left( \hat{y}_i - y_i \right) \mathbf{x}_i

Prediction minus target, times the input. Identical in form to linear regression's gradient, which is not a coincidence. Both are generalised linear models.

One prediction, in order

Take the dot product of the weights with the features and add the bias.
Push that scalar through the sigmoid.
Read the result as the probability of the positive class.
Compare it to a threshold to get a decision.
Choose that threshold from the cost of each error, not by defaulting to 0.5.

Why it is still worth knowing

It is the last layer of almost every classifier ever built, including every neural network that ends in a sigmoid or a softmax.

Understanding it is understanding what that final layer is doing, and why its output can be read as a probability at all.

Test yourself

One question on this card. Every option explains itself once you pick, right or wrong.

Question 1

Why does logistic regression carry the word regression when it classifies?

Machine Learning

55 concepts in this module, then a graded test with every answer explained.

Study this module →