Start studying
Module 10 · Experimentation Module contents
Module 10  ·  Card 3 / 13intermediate

Unit of Randomisation

The unit of randomisation is the thing assigned to a group: a user, a session, a page view or a whole region.

Each unit must always see the same version, so assigning by user keeps one person's experience consistent.

For example, randomising by page view could show one shopper 2 different prices within an hour, which confuses them.

Units must not affect each other, so a change to a messaging app may need randomising by friend group rather than by person.

The analysis should use the same unit as the assignment, or the standard errors come out too small.

Learn in depth ↓
Module 10 · Experimentation · 13 concepts
In depth  /  15 lines

Unit of Randomisation, in more detail

Choosing the unit of randomisation decides what counts as one independent observation in a test.

The common units are users, sessions, page views, devices, stores and regions.

The right choice depends on what the change affects and how users experience it.

User-level assignment is the usual default, because each person sees one consistent version.

It needs a stable identifier, such as a login, since a cleared cookie can put the same person into a different group.

Session or page view assignment gives more units and faster tests, but suits only changes users will not notice between visits.

Units can affect each other, which breaks the test: a new feature in a chat app changes the experience of friends in the control.

This spillover is called interference, and it usually makes the measured effect too small.

Larger units, such as whole cities or friend groups, keep the spillover inside each unit.

The cost is fewer units, so tests on cities or stores need longer runs or larger effects.

Marketplaces are a classic case, since a change that helps buyers in treatment can take sellers away from buyers in control.

The analysis must treat each unit as one observation, not each page view inside it.

Counting page views as independent when users were assigned overstates precision and gives too many false positives.

In practice, teams record the unit in the test plan and check that every metric is calculated at that level.

The usual mistake is assigning by user but working out standard errors as if every page view were independent.

Test yourself

One question on this card. Every option explains itself once you pick, right or wrong.

Question 1
A test of 2 prices assigns each page view to a version at random. What is the main problem?
Experimentation

13 concepts in this module, then a graded test with every answer explained.

Study this module →
↑↓ move↵ openesc close