Novelty and primacy effects are about how users react to change itself, separate from the change's real value.
They matter most for returning users, who notice that something is different.
New users have no old version to compare with, so they react only to the design in front of them.
Novelty inflates early results: a new button gets clicked because it is new.
Primacy depresses early results: users slow down while they relearn where things are.
Both fade, but at different speeds for different changes and different users.
Plotting the daily treatment effect shows the pattern, either shrinking from a high start or rising from a low one.
Splitting results by new and returning users is a strong check, since an early swing among returning users only points to the change itself.
Running a test for several weeks lets early reactions settle before the decision.
Longer tests cost time, and more users drop out or clear cookies the longer a test runs.
Measuring only users who joined after the test began removes both effects but takes longer to fill.
A long-term holdout, kept on the old version for months, shows whether the effect lasts.
In practice, visible changes such as redesigns are the most prone, while changes users cannot see rarely show either effect.
Teams often discount the first week of a visible test or leave it out of the decision.
The usual mistake is launching on the first week's lift, which then fades and never shows up in the business numbers.