Before you trust last year's A/B test wins, re-run the math

Most testing programmes report more winners than they actually produced. Before last year's wins shape this year's roadmap, re-check each one: did it reach the sample size set before launch, run in whole weeks, keep an even traffic split, and still show up in revenue after it shipped? Why the list of wins is usually inflated The companies with the most experience report modest hit rates. Writing in Harvard Business Review, Ron Kohavi and Stefan Thomke describe how only about a third of ideas tested at Microsoft improved the metric they were designed to improve. Thomke, who studied Booking.com's programme closely, says that on average only around one experiment in ten yields a result that improves the bottom line. So if a programme's deck shows six winners out of ten, either it is better than Microsoft, or the bar for "winner" was lower. The second is far more common. The usual causes are tests called early, tests read against several metrics until one looked good, and traffic splits that broke without anyone noticing. Re-running one takes ten minutes Take a test that was reported as a 12% win. Each version saw 20,000 visitors. The control converted 600 times (3.0%) and the variant 672 times (3.36%). Re-run the numbers and the result is genuinely significant, with a p-value of about 0.04. But the range the true lift probably sits in runs from roughly 0.5% to 23%. The headline said 12%. The honest read is "somewhere between almost nothing and a lot". Then check question one. To detect a 10% lift from a 3% baseline at 95% confidence and 80% power, each version needs around 53,000 visitors. If the team planned for that, this test stopped well short. If nobody wrote a plan, there is no way to know whether it stopped because it reached its sample or because the graph looked good that afternoon. It may still be a real winner. It is not yet a proven one, and it should not anchor a roadmap. Why early winners shrink after launch Even a test run correctly tends to overstate its own result. A variant gets declared the winner partly because it is good and partly because random variation happened to flatter it during the test window. Ship it, and the flattering part does not come along. The lift you see in the following months usually sits below the number in the report, sometimes well below. That is not a flaw in testing. It is a reason to plan revenue around the low end of the uplift range rather than the headline, and to treat any reported lift above 20% on a mature page with suspicion until it has held for a month in production. What to do with the tests that fail Relabel them inconclusive, not lost. An inconclusive test is not a loser and it is not a quiet winner either. It is an open question, and treating it as answered is how roadmaps drift. Do not rip out shipped changes automatically. Most of them do no harm. Re-run the few that underpin big decisions, such as pricing, shipping thresholds or subscription defaults, ideally as a holdout where a slice of traffic keeps seeing the old version for a few weeks. Then change the rule going forward. The sample size gets written down before launch, the primary metric gets chosen before launch, and the result gets read once, when the sample is reached. What a win that survives the audit looks like The tests worth trusting tend to be judged on money rather than on conversion rate alone. One shipping test I ran for Momentous added $2 million in profit. The roadmap behind it was built around business metrics, profit, order value and subscription take rate, which is why that number stands up to scrutiny. Most programmes never get there because the backlog is full of results nobody re-checked, and the next round of tests gets prioritised on top of them. I keep a free significance and sample size calculator open for exactly this kind of audit. It also checks the traffic split for sample ratio mismatch, which is the question most audits skip.