A/B tests usually fail on slow eCommerce sites for a simple reason. The site underneath is dragging every variant down to the same number, so neither version gets a fair chance to win.
A test that runs for six weeks and comes back flat still costs you six weeks. It took a slot on the roadmap, somebody built both versions, and it chipped away at everyone’s patience with CRO. Run enough of them and the business decides testing doesn’t work. More often, the foundations weren’t ready for it.
This guide covers why speed caps your results, what to fix before you test, and which changes don’t need a test at all.
Why does a slow site make A/B tests come back flat?
If your product page takes six seconds to load on a mid-range phone, a sharper headline won’t rescue it. A chunk of your visitors have gone before either version has finished rendering. The ones who stay are the most determined, and they were probably going to buy whatever you showed them.
So the test measures the patient few, finds no difference between them, and you blame the idea.
Speed isn’t a small effect either. A study by Google, 55 and Deloitte, Milliseconds Make Millions, looked at 37 brand sites and over 30 million sessions. A 0.1 second improvement in mobile site speed lifted retail conversion rates by 8.4%. If a tenth of a second does that, a page that’s seconds slower than it should be is hiding more than most headline tests will ever find.
What else goes wrong on a slow site?
Heavy themes and stacked third-party scripts cause a second problem. Testing tools often load the original page first and then swap in the variant, so the visitor sees the old version flash up for a split second before it changes. That flicker makes the variant look broken and skews the result against it.
Slow pages also make tests take longer. Fewer visitors reach the part of the page you’re testing, so you need more traffic or more weeks to reach a reliable answer.
What should you fix before you start testing?
We treat speed and frontend architecture as phase one of CRO, before a single test runs. It’s less exciting than a testing roadmap. It also works.
Start with the pages your traffic actually lands on.
- Check mobile load times on your top five landing pages. Google’s web.dev guidance suggests the main content should appear within 2.5 seconds for most visits.
- Count the scripts firing on your product page, and ask what each one earns.
- Remove apps and extensions nobody has used in months.
- Watch ten session recordings of people who left without adding anything to the basket.
- Fix the obvious problems those recordings show you.
The Skirting Board Shop shows the order working. We rebuilt the site on Hyvä, then spent a year on CRO on top of it. Conversion went up 51% and new visitors went up 670%. The tests in that year had a fast site to work with, so a good idea could actually show up in the numbers.
Helmet City followed the same order. The rebuild took four seconds off the load time and conversion went up 16%. Only then did the retained CRO work start landing, with another 9% on top.
How do you know if speed is capping your tests?
You don’t need a full audit to get a first answer. Open PageSpeed Insights and run your top product page. The section at the top shows how real visitors experienced the page over the last 28 days, if Google has enough data for your site. If the main content takes longer than 2.5 seconds for a large share of mobile visits, speed is a likely cap.
Then look at your own numbers. In GA4, compare conversion rate on mobile and desktop for the same pages. A big gap is normal. A gap that has widened since your last theme or app change is a sign something got heavier.
Finally, check what your testing tool adds. Some load a large script before the page can show anything. Ask your agency or the vendor whether it’s loading asynchronously and how much it weighs.
How much traffic do you need to run a test?
More than most people think. A test needs enough visitors and enough orders in each version to tell a real difference from chance. A page with a few hundred sessions a week and a handful of orders will take months to reach a reliable answer on anything but a huge change.
Your testing tool will have a sample size calculator. Put in your current conversion rate and the smallest lift you’d care about. If the answer is longer than eight weeks, test something on a busier page, or make the change and measure it before and after instead.
Which changes don’t need an A/B test?
Some fixes don’t need six weeks of traffic to prove them. A filter that returns nothing, a delivery charge you only find at checkout, a search box that can’t handle your best seller’s name spelt wrong. Fix those and move on. We’ve written up the CRO fixes that take a day separately.
Testing the obvious is how agencies fill a retainer. We’d rather the hours went where you genuinely need evidence.
What is worth testing?
Save testing for decisions that are uncertain and expensive to get wrong.
| Worth testing | Just fix it |
|---|---|
| How you show price on bundles | A broken or empty filter |
| How much of the range to show on a category page | Delivery cost hidden until checkout |
| Whether free delivery starts at £50 or £75 | Search failing on common misspellings |
| Long-form vs short-form product copy | A slow page or a broken image |
| Subscription vs one-off as the default option | A pop-up blocking mobile checkout |
What does a run of flat tests tell you?
If three or four tests in a row come back inconclusive, stop testing for a fortnight and look at the site instead. Flat results in a row are usually a sign that something underneath is capping every variant, whether that’s speed, broken tracking or too little traffic reaching the page.
Fix what that turns up and the tests start winning again. Then the conversation changes. Instead of arguing about whether CRO is worth it, you’re asking whether the warehouse can cope with the next promotion.
Frequently asked questions
How long should an A/B test run?
Long enough to cover at least one full weekly cycle, and long enough to reach the sample size your testing tool says you need. Stopping early because one version is ahead is the most common way to get a false winner.
Does page speed really affect conversion that much?
Yes. In Google’s study with 55 and Deloitte, a 0.1 second improvement in mobile speed lifted retail conversion by 8.4%. Our own clients have seen bigger lifts from rebuilds that took seconds off, not tenths.
What is test flicker?
It’s when visitors briefly see the original page before the testing tool swaps in the variant. It’s more common on heavy, slow pages, and it biases results against the variant.
Should we stop testing altogether if the site is slow?
Pause, fix the speed, then restart. Testing on a slow site wastes traffic you could be using once the foundations are right.
What if we don’t have enough traffic to test?
Then most changes should be judged on evidence rather than tests: session recordings, customer feedback and before-and-after numbers. Keep testing for the few decisions where the stakes are high.
How we can help
Send us your last three test results and the pages they ran on. We’ll tell you whether the site was holding them back and what we’d fix before the next one. If you’d rather start with your own numbers, our Free CRO Review runs your live speed data and shows you the first five issues.
