Naive Bayes in machine learning

Module 01 · Machine Learning · 55 concepts

In depth / 15 lines

Naive Bayes, in more detail

Naive Bayes counts rather than optimises, which is why it is so fast.

It works out how often each feature turns up with each class.

Then it multiplies those numbers together to score a new example.

The naive part is assuming the features are unrelated to each other.

In text that is obviously false, since words arrive in phrases.

It still works, because the ranking often survives the wrong assumption.

It needs very little data before it becomes useful.

That makes it sensible when examples are scarce.

A word never seen with a class would give a probability of zero.

One zero wipes out the whole product, so a small count is added everywhere.

Its probabilities are usually badly calibrated and far too confident.

Use the ranking it gives you, not the number itself.

It trains almost instantly, even on very large collections of text.

It stays a reasonable baseline for spam filtering and topic sorting.

If it beats your neural network, your neural network has a problem.

Want the full technical article? Wikipedia ↗

Deep dive

What is underneath Naive Bayes

Bayes' theorem, applied to a label

P(y \mid \mathbf{x}) = \frac{P(\mathbf{x} \mid y) \, P(y)}{P(\mathbf{x})}

$P(y \mid \mathbf{x})$ : the posterior: what you want, the probability of the class given the features.
$P(\mathbf{x} \mid y)$ : the likelihood: how probable these features are if the class were $y$ .
$P(y)$ : the prior: how common that class is at all.
$P(\mathbf{x})$ : the evidence, which is the same for every class.

Because the denominator does not depend on $y$ , it can be dropped when you only need the winner.

\hat{y} = \arg\max_{y} \; P(\mathbf{x} \mid y) \, P(y)

The naive assumption, stated plainly

$P(\mathbf{x} \mid y)$ over $d$ features is a joint distribution, and estimating it needs a number of examples exponential in $d$ .

P(\mathbf{x} \mid y) = \prod_{j=1}^{d} P(x_j \mid y)

The assumption is that every feature is conditionally independent of every other one, given the class.

That turns one impossible estimate into $d$ easy ones, each a simple count.

It is also flatly false. In text, "New" and "York" are anything but independent, and the model counts that pair as two separate pieces of evidence.

Why it works anyway

Classification only needs the argmax to be right, not the probabilities.

Double-counting correlated evidence pushes the winning probability towards 1 and the rest towards 0, so the probabilities become badly overconfident. The ranking usually survives.

That is the honest summary: naive Bayes is a poor probability estimator and a surprisingly good classifier, and you should never quote its confidence.

Logs, and the smoothing that stops a zero

\log P(y \mid \mathbf{x}) \propto \log P(y) + \sum_{j=1}^{d} \log P(x_j \mid y)

Multiplying hundreds of small probabilities underflows to zero in floating point, so everything is done in logs and the product becomes a sum.

P(x_j \mid y) = \frac{\mathrm{count}(x_j, y) + \alpha}{\mathrm{count}(y) + \alpha \, |V|}

$\alpha$ : the smoothing constant, usually 1, which is called Laplace smoothing.

Without it, a single word never seen with a class gives that class a probability of exactly zero, and one unseen word vetoes all the other evidence.

One prediction, in order

Start with the log prior of each class.
For each feature, add the log of its smoothed conditional probability.
Do that for every class.
Take the class with the largest total.
Do not report the resulting probability as a confidence.

Test yourself

One question on this card. Every option explains itself once you pick, right or wrong.

Question 1

What exactly is the naive assumption?

Machine Learning

55 concepts in this module, then a graded test with every answer explained.

Study this module →