Start studying
Module 9 · Sampling and Inference Module contents
Module 9  ·  Card 5 / 16beginner

Sampling Bias

Sampling bias happens when the way a sample is chosen makes some members of the population more likely to be included than others.

The sample then differs from the population in a systematic way, so its estimates are off in one direction.

For example, a phone survey run only during working hours under-represents people who work those hours.

Collecting more data the same way does not fix it, because a bigger biased sample is just as biased.

Voluntary response surveys are a common source, since people with strong opinions are the most likely to reply.

Learn in depth ↓
Module 09 · Sampling and Inference · 16 concepts
In depth  /  15 lines

Sampling Bias, in more detail

Sampling bias is a problem in how a sample is chosen, not in how the data is analysed afterwards.

It makes some members of the population more likely to be included, so the sample leans in one direction.

No method applied later can fully remove it, because the people who were missed left no data.

Undercoverage happens when part of the population is missing from the list, such as people without internet access in an online survey.

Non-response bias happens when the people who decline to take part differ from those who agree.

Voluntary response bias happens when people choose to join, so the most motivated dominate.

Survivorship bias happens when only the survivors of some process are visible, such as studying only companies still trading.

For example, studying only successful start-ups makes risky strategies look safer than they are.

Convenience samples, such as surveying the people in one office, are quick but rarely represent anyone else.

Bias does not shrink with sample size, unlike random error, which falls as the sample grows.

A huge biased sample is dangerous because its apparent precision makes a wrong answer look trustworthy.

Checking the sample against known population figures, such as age or region, reveals some bias but not all.

Weighting the sample to match known figures can reduce bias in those variables, but not in ones nobody measured.

In practice, describing exactly who was asked and who answered is the best protection for readers.

The common misconception is that a large sample cannot be biased, when size and bias are separate problems.

Want the full technical article? Wikipedia ↗
Test yourself

One question on this card. Every option explains itself once you pick, right or wrong.

Question 1
A news site's online poll gets 50,000 votes, with 80% against a new law. Why is this weak evidence of public opinion?
Sampling and Inference

16 concepts in this module, then a graded test with every answer explained.

Study this module →
↑↓ move↵ openesc close