The multiple testing problem appears whenever many hypotheses are tested and only the significant ones get noticed.
Each test has a small chance of a false alarm, and those chances build up across many tests.
With enough tests, finding something significant becomes almost certain even when nothing is real.
At a 0.05 threshold, the chance of at least 1 false alarm in n independent tests is 1 minus 0.95 to the power n.
For 10 tests that is about 40%, and for 100 tests it is over 99%.
Extra tests arrive in many forms: many metrics, many customer segments, many time windows or many versions of an analysis.
The Bonferroni correction keeps the overall false alarm rate at 5% or below but makes each test much stricter.
Its strictness costs power, so real effects among many tests become harder to find.
The Benjamini Hochberg method instead controls the false discovery rate, the expected share of significant results that are false.
It suits exploratory work with many tests, where a few false findings among many true ones are acceptable.
P-hacking is testing many variations and reporting only the ones that worked, often without meaning to.
Choosing measures, cut-offs or subgroups after seeing the data all count as hidden extra tests.
Writing down the main hypothesis and analysis before the data arrives, called pre-registration, is the strongest defence.
In practice, teams name 1 primary measure in advance and treat everything else as a lead to check later.
The common misconception is that a significant result among many tests is as strong as the same result from a single planned test.