A/B testing brings the logic of a controlled experiment to products, websites and businesses.
It exists because most changes have small effects, and small effects are invisible in ordinary before and after numbers.
Seasons, marketing and news move metrics all the time, and only a comparison run at the same moment removes them.
A test starts with a hypothesis, such as a shorter form raising sign-ups, and a primary measure of success.
Users are split at random, each group sees one version, and both groups run over the same days.
At the end, the difference between the groups is compared with the variation expected by chance.
If the difference is significant and large enough to matter, the change is launched.
Many tests find no meaningful effect, which is useful too, because it stops wasted effort on ideas that do not work.
A/B tests need enough traffic, so small sites may wait months for an answer about a small effect.
They measure short-term behaviour well, but long-term effects such as loyalty are harder to capture.
Some changes cannot be tested fairly, such as a price change that customers will discuss with each other.
Ethics matters too: testing button colours on users is accepted, but testing anything that could harm them is not.
In practice, large online companies run many tests at once, and a large share of ideas fail to improve anything.
Writing down the hypothesis, metric, sample size and run time before starting makes a test far harder to fool.
The common misconception is that an A/B test shows why a version won, when it only shows that it did.