Start studying
Module 10 · Experimentation Module contents
Module 10  ·  Card 11 / 13advanced

Interaction Effects in Experiments

An interaction effect happens when the effect of one change depends on another change running at the same time.

Large companies run many tests at once, so the same user can be in several experiments.

For example, a new font and a new colour scheme might each help on their own but clash when a user sees both.

Most overlapping tests do not interact, so running them side by side is usually safe and much faster.

Where interaction is likely, tests are kept apart or combined into one test that tries every combination of versions.

Learn in depth ↓
Module 10 · Experimentation · 13 concepts
In depth  /  15 lines

Interaction Effects in Experiments, in more detail

Interaction effects matter because modern products often run many experiments on the same users at once.

Running tests side by side is far faster than queuing them, so teams want to know when it is safe.

An interaction means the combined effect of 2 changes differs from the sum of their separate effects.

For example, a larger image and a longer description might each raise sales but together push the buy button off screen.

Independent assignment means each test is randomised separately, so every combination of versions appears.

With independent assignment, each test's average effect is still measured fairly, even if interactions exist.

Interactions can be checked by comparing results within each combination of versions.

Detecting an interaction needs far more data than detecting a single effect, so small ones go unnoticed.

In practice, strong interactions between unrelated tests appear to be uncommon.

Tests that touch the same part of the page, or the same decision, are the most likely to interact.

Those tests can be kept mutually exclusive, so no user sees both.

A factorial test combines changes on purpose, testing every combination in one design.

For 2 changes with 2 versions each, that gives 4 groups and shows both the separate and the combined effects.

Teams often group tests into layers, where tests in the same layer never share users.

The usual mistake is fearing all overlaps and running tests one at a time, which slows learning far more than interactions cost.

Want the full technical article? Wikipedia ↗
Test yourself

One question on this card. Every option explains itself once you pick, right or wrong.

Question 1
2 teams test a new font and a new colour scheme at once. Each wins alone, but users who see both convert worse. What is this?
Experimentation

13 concepts in this module, then a graded test with every answer explained.

Study this module →
↑↓ move↵ openesc close