Running an experiment is the easy part. Reading it correctly is where the value is created or thrown away, and most teams throw it away in the same handful of ways — stopping when the line looked good, celebrating a number the sample never supported, or shipping a variant that did nothing because somebody preferred it.
The frustrating thing is that a badly read experiment is worse than no experiment. It produces a decision with a number attached, which is far harder to argue with than an opinion, and it makes the whole programme less trustworthy the third time it happens.
This is the checklist I apply to a finished test before acting on it: what the numbers mean, what to verify, what to do with each outcome, and the failure modes that look like wins. It is the decide stage of the conversion loop, done properly.
What do the numbers actually mean?#
Less than the interface implies, and the phrasing matters.
A tool reporting "92% probability to beat baseline" is not saying the variant is 92% better, and a p-value of 0.04 is not saying there is a 4% chance you are wrong. Both are statements about how surprising your data would be under a specific assumption, and the gap between that and what people hear is where most misreadings begin.
| Reported | What it means | What people hear |
|---|---|---|
| p = 0.04 | Data this extreme is unlikely if there were no effect | A 4% chance of being wrong |
| 95% confidence | The method captures the true value 95% of the time | A 95% chance this result is right |
| +12% uplift | The point estimate from this sample | Conversion will rise 12% |
| Interval −2% to +26% | The effect is plausibly anywhere in here | (Usually not read at all) |
The bottom row is the one that matters most and gets displayed least prominently. A headline uplift of 12% with an interval spanning −2% to +26% is a result that is entirely consistent with the change doing nothing, and shipping it on the strength of the headline is shipping on noise.
Read the interval, not the point estimate#
If the range includes zero, the honest summary is "we cannot tell". If the whole range is positive but the lower bound is 1%, the honest summary is "probably a small win". Both are useful and neither is the number on the card.
The reported uplift is an overestimate#
This is not intuitive and it is important. Tests that reach significance tend to do so partly because the sample happened to favour the variant, so the measured effect is systematically larger than the real one. Expect the effect in production to be smaller than the result claimed, and budget accordingly.
Did the test run properly?#
Check this before reading the result, because a broken test produces a number regardless.
Was the sample reached?#
The size calculated before starting, not the size that happened to accumulate. A test stopped at 60% of its target has a wider interval than the tool's headline suggests and a much higher false-positive rate, whatever the display says.
Did it run for at least two full weeks?#
Behaviour differs by day of week, and payday cycles are real. A test covering nine days has measured a specific slice of the month, and if that slice included a campaign or a bank holiday it has measured the campaign.
Was the split actually even?#
A 50/50 test that delivered 52/48 traffic is a sample ratio mismatch, and it usually means something is wrong with assignment — a bot filter, a caching layer, a redirect. That is a reason to discard the result rather than to interpret it.
Did both variants perform normally otherwise?#
Compare the pages on things the test should not have affected — load time, error rate, bounce on unrelated pages. A variant that was 400ms slower was testing your idea plus a slower page, and on mobile the slowdown can be the whole effect. Speed acts as a floor and it contaminates results quietly.
Is the win real?#
Four checks that separate a finding from a coincidence.
Does the mechanism make sense?#
A result you cannot explain is a result you should not trust much. If moving a button lifted conversion, that is plausible. If changing a footer colour lifted checkout completion by 15%, something else happened and it is worth finding out what.
Did the secondary metrics hold?#
A change that lifts add-to-cart while depressing completion has moved the problem downstream, not solved it. Revenue per session is the honest scoreboard — a conversion-rate win with a lower average order value is frequently a loss.
Does it survive segmentation?#
Split the result by device and by new versus returning visitors. A flat overall result concealing a strong mobile win and a desktop loss is common and changes the decision entirely. So is a "win" driven entirely by one traffic source that was running an unrelated campaign.
Would you have believed the opposite?#
The most useful question in the list. If the result had gone the other way, would you have accepted it as readily, or gone looking for reasons the test was flawed? An honest answer usually reveals whether you are reading or confirming.
What do you do with a flat result?#
Revert, and record it — this is the most common outcome and the most mishandled.
Flat means the change did not matter at the scale you can detect. That is genuinely useful: it tells you the thing you thought was blocking people was not, which redirects effort better than another small win would. Treating it as a failure of the experiment rather than an answer is what makes teams stop running them.
Revert to the simpler version#
If two versions perform identically, keep the one that is less code, fewer requests and easier to maintain. Complexity that buys nothing should not survive on the grounds that it already exists.
Do not ship it because somebody prefers it#
If the variant did nothing and a stakeholder likes it, that is a taste decision and should be called one. It may well be the right call — brand consistency is a real reason — but describing it as a conversion improvement is how the numbers stop meaning anything internally.
Ask whether the test was too small to detect the effect#
A flat result on an underpowered test means "we could not tell", not "there is no effect". If the interval spans −8% to +9%, a genuine 3% improvement was never detectable, and concluding the change does not work is overreading the data.
Consider what else changed that fortnight#
A test does not run in isolation. A price change, a stock-out on a popular line, a campaign sending different traffic, or a competitor’s promotion all move conversion during the window and none of them appear in the tool. Writing down what else was happening turns an unexplained result into an explained one.
What do you do with a loss?#
Learn from it, because losses generalise better than wins.
A variant that lost has told you something true about your customers that you believed otherwise, and that belief was probably applied elsewhere on the site. Finding out that adding urgency messaging depressed conversion is more valuable than another confirmation that a bigger button helps, because it changes what you do next in several places.
Check it was not an implementation bug#
A large unexplained loss is often a broken variant rather than a bad idea — a button that failed on Safari, an image that did not load, a script error on one device class. Look at the error rate before concluding anything about the hypothesis.
Write down what it disproved#
Specifically. "Adding a countdown timer to the product page reduced completed orders by 4%" is a finding somebody can act on in a year. "The urgency test failed" is not.
Losses are why the log exists#
A record of only wins is a marketing document. The failures are what stop a new colleague re-proposing in November the test that already lost in March, and they are the half of the record that nobody keeps.
When can you trust a small win?#
When it is cheap to keep, consistent across segments, and you are not stacking it on other unverified small wins.
A 2% lift that clears significance on a well-run test is probably real and probably smaller than 2%. If the change costs nothing to maintain, ship it. The problem is not any individual small win; it is a programme that has claimed fourteen of them and cannot find the 30% compound improvement they imply.
Beware the accumulation fallacy#
Small wins do not add up the way a spreadsheet suggests. They interact, they decay, and some were false positives that nobody revisited. Reporting a cumulative uplift by multiplying individual results out is a number that will not survive contact with the actual conversion rate.
Re-test the important ones#
If a result is going to drive significant investment, run it again. Replication is unglamorous and it is the only real defence against a false positive, and the second run costs a fraction of the decision it protects.
Check the effect persists#
Novelty effects are real: returning visitors respond to a change partly because it is new, and that response fades. A win measured in week one that has vanished by week six was a reaction to change rather than an improvement, and only looking again reveals it.
What are the common ways to be wrong?#
Five, and they account for most bad decisions in conversion work.
- Stopping early — checking daily and calling it when the line looks good, which manufactures winners from noise.
- Choosing the metric afterwards — something always moved, so any test can be a win if you pick the number last.
- Testing many things at once — twenty simultaneous comparisons will produce a "significant" result by chance alone.
- Ignoring the interval — reading the point estimate as the effect and shipping on a range that includes zero.
- Not segmenting — an aggregate that hides opposite effects on mobile and desktop.
The first is by far the most common and the easiest to fix, because it requires no statistics — only the discipline to write down the duration in advance and honour it.
Multiple comparisons deserve their own warning#
If you measure eight metrics on one test, one of them will look significant at the usual threshold by chance. That is not a subtle statistical point; it is arithmetic. Naming one primary metric before starting is what protects you from it.
How do you report a result?#
In a short, honest format that includes what you did not learn.
- The hypothesis as it was written before the test.
- What was built, in one sentence.
- Sample and duration — planned and actual.
- The result, with the interval, not only the point estimate.
- Segment breakdown by device at minimum.
- The decision and, in one line, what it means for what you do next.
Ten minutes per test. The value is entirely in a year's time, when somebody asks whether you have tried something and the answer takes thirty seconds instead of a week.
Say what you could not detect#
A line stating the smallest effect the test could have found is worth including, because it distinguishes "this does nothing" from "we could not tell". Those two conclusions lead to very different next steps.
Write for the person who was not there#
In eighteen months everybody in the room will have partially forgotten and some will have left. A result that needs its author present to interpret is a result that will be re-run, which is the cost the log exists to avoid — the same argument as documenting anything else non-obvious.
When should you stop testing?#
When three consecutive tests come back flat, which is usually the signal that the cheap wins are gone.
Conversion work has a ceiling. Once the obvious leaks are closed, remaining gains get small, slow and statistically fragile, and a team still grinding out tests at that point is spending real money on decimal points while the larger levers go untouched.
The bigger levers are harder conversations#
Price, product range, delivery proposition and traffic quality all move conversion more than any page change, and none of them are a developer's decision. A conversion programme that never escalates to those is optimising within a constraint nobody has examined.
Return to the funnel, not to the backlog#
Rather than working further down a list of hypotheses about one page, re-run the funnel measurement. The leak has often moved, and the next round of work belongs somewhere else entirely — which is where the loop restarts.
What does it cost?#
An hour per result to read properly, against weeks of work spent acting on one read badly.
Checking the sample, the duration, the split, the interval and the segments is genuinely under an hour. Writing the record is ten minutes. That is a very small tax on a test that took three weeks to produce, and it is the step most often skipped because the result is already on screen and everybody wants to move.
The honest counterweight: rigour has a cost too, and it is possible to be so careful that nothing ever ships. Most stores do not have the traffic to resolve a 3% effect, and a team that refuses to act without significance on a store like that will act on nothing at all. The correct response to low traffic is not more statistical caution — it is to ship well-established improvements without testing them and reserve experiments for the genuinely uncertain and genuinely expensive decisions.
A badly read experiment is worse than no experiment. It produces a decision with a number attached, which is much harder to argue with than an opinion.
Conclusion#
Read the interval rather than the headline. A +12% uplift with a range from −2% to +26% is entirely consistent with the change doing nothing, and the reported effect is systematically larger than the real one because tests reach significance partly when the sample happened to favour the variant.
Verify the test before interpreting it: was the calculated sample reached, did it run two full weeks, was the traffic split actually even, and did both variants behave normally on things the test should not have touched. A variant 400ms slower was testing your idea plus a slower page.
Then check whether the win is real — does the mechanism make sense, did secondary metrics hold, does it survive segmentation by device and visitor type, and would you have accepted the opposite result as readily. Revenue per session is the honest scoreboard, not conversion rate.
Treat flat as an answer rather than a failure: revert to the simpler version, record what it disproved, and do not let a preference be reported as a conversion improvement. Treat losses as the valuable half, after ruling out an implementation bug, because they generalise better than wins and nobody writes them down.
Watch the five failure modes — stopping early, choosing the metric afterwards, testing many things at once, ignoring the interval, and not segmenting. And know when to stop: three flat results in a row usually means the cheap wins are gone and the remaining levers are price, product, delivery and traffic quality, which are harder conversations than any page change. If you want the programme run with that discipline built in, that is the CRO work.