Power is the chance that a test finds an effect that really exists, so it measures how sensitive a test is.
Low power wastes effort, because a study can run for weeks and still be unable to detect the effect it was built for.
Checking power before a test starts is how teams avoid running studies that cannot succeed.
Power depends on 4 things: the sample size, the true size of the effect, the noise in the data and the significance threshold.
A bigger sample raises power, though each extra user adds less as power approaches 100%.
Larger effects are easier to detect, so a 10% lift needs far fewer users than a 1% lift.
The sample needed grows with 1 divided by the effect squared, so halving the effect needs about 4 times the data.
Less noisy measures, such as a conversion rate instead of revenue per user, often need smaller samples.
A power of 80% is a common target, meaning a 1 in 5 chance of missing a real effect of the planned size.
Studies with low power have a hidden second problem: the effects they do detect are usually overestimates.
That happens because only the samples that happened to show a large effect cross the significance threshold.
Power calculations need an assumed effect size, which is often an educated guess based on past results.
Power worked out after the fact from the observed effect adds nothing beyond the p-value and is best avoided.
In practice, power analysis sets the sample size and the run time of an experiment before it starts.
The usual mistake is reading a non-significant result from a small study as evidence that the effect is absent.