Every test can go wrong in 2 opposite ways, and naming them helps decide which is worse in a given case.
A Type I error is a false positive: acting on an effect that is not there.
A Type II error is a false negative: missing an effect that is there.
The Type I error rate is set directly by the significance threshold, called alpha.
With alpha at 0.05, about 5% of tests where the null is true still come out significant.
The Type II error rate, called beta, depends on the sample size, the true size of the effect and the noise in the data.
Medical screening is an everyday example: a false alarm worries a healthy patient, and a miss leaves an illness untreated.
Which error is worse depends on cost: a false alarm on a cheap change matters less than one on a costly rebuild.
Lowering alpha to 0.01 cuts false alarms but makes real effects harder to detect.
The only way to cut both at once is more data, or less noise in it.
The share of false alarms among significant results is a different number from alpha and can be much higher.
When few of the ideas being tested are real, many of the significant results can be false alarms.
In practice, teams choose both acceptable error rates before a test starts, based on the cost of each mistake.
Writing down the cost of each error first keeps the choice of threshold from being arbitrary.
The common misconception is that a 5% threshold means only 5% of significant findings are wrong, when that share can be far higher.