Guardrail metrics are measures that must not get worse, checked alongside the primary metric in every test.
They catch changes that win on the goal but cause damage somewhere else.
For example, a pop-up may raise sign-ups while page load time and unsubscribe rates get worse.
Typical guardrails include page speed, error rates, refunds and complaints.
A test that wins on its primary metric but breaks a guardrail is usually not launched as it stands.
Guardrail metrics protect the things a test is not trying to improve but must not damage.
They exist because a primary metric only measures what it measures, and a change can win there while harming everything else.
A short, fixed set of guardrails, checked in every test, builds a habit of looking for harm.
Some guardrails protect the business, such as revenue, refunds and cancellations.
Some protect users, such as complaints, unsubscribes and accessibility measures.
Some protect the product, such as page load time, crash rates and error rates.
A breach is usually defined in advance as a change beyond a set size, such as pages loading more than 5% slower.
Guardrails often use 1-sided checks, because only a change for the worse matters.
Trust guardrails check the test itself, such as a balanced group split and complete logging.
Too many guardrails bring back the multiple testing problem, so false alarms on them become likely.
Very sensitive guardrails can block good changes over tiny, unimportant shifts.
A breach does not automatically kill a change, but it forces a deliberate decision about the trade-off.
In practice, a breach often points to a fixable side effect, such as a slow image, rather than a bad idea.
Fixing the side effect and testing again is common and usually quick.
The usual mistake is checking guardrails only when the primary result is disappointing, rather than in every test.
One question on this card. Every option explains itself once you pick, right or wrong.
13 concepts in this module, then a graded test with every answer explained.