A/B testing is the respectable face of conversion work, so it gets proposed constantly and run rarely well. The obstacle is arithmetic: most B2B sites do not produce enough conversions for a test to say anything trustworthy in a reasonable time.

The dangerous part is that this does not announce itself. An underpowered test still shows a winner, still shows a percentage uplift, and is still wrong.

The arithmetic, roughly

The number of conversions you need per variant rises steeply as the effect you are trying to detect gets smaller. For a baseline conversion rate of a few per cent, detecting a relative improvement of twenty per cent takes on the order of hundreds of conversions per variant. Detecting five per cent takes thousands.

Turn that into your own numbers. If the page converts fifty times a month and you split traffic in half, each variant sees twenty-five. To reach a few hundred per variant you are running the test for the better part of a year – during which the season changes, the ad mix changes, and the thing you are measuring is no longer the thing you started measuring.

Use a sample size calculator before designing the test, not after. If the required duration exceeds about six weeks, the test is not viable and running it anyway is worse than not running it.

What to do instead

Below the volume threshold, these produce more reliable improvements than testing does, and considerably faster.

  1. Fix the technical friction. Page speed on the template that carries the traffic, layout shift moving the button as it is tapped, validation that rejects a valid phone number, a form that loses its contents on error. These are defects, not hypotheses, and they do not need testing to justify fixing.
  2. Watch five people use it. Five sessions of moderated testing will surface more than a quarter of A/B tests on a low-traffic site. It costs a day.
  3. Read the funnel by segment. Aggregate conversion rate hides the problem. "Mobile users on the shipping step" is a solvable problem; "our conversion rate is low" is not.
  4. Read the support tickets and sales calls. The questions people ask before buying are the information missing from the page. This is the cheapest research available and almost nobody uses it.
  5. Fix the information gaps. Price, delivery time, minimum order, whether you serve their country. Missing decision information sends people to a competitor to find it, and they often stay there.

When testing does make sense

High-volume e-commerce, transactional funnels with thousands of conversions a month, or a page receiving heavy paid traffic where a small percentage difference is worth real money.

Even then, test things that could plausibly move the number by a lot. Testing button colours on a page with adequate volume is a valid experiment and a poor use of it; testing whether the checkout has three steps or one is worth the traffic you spend on it.

And test one meaningful change at a time. A redesign of the whole page tested against the original tells you the new page is better or worse, and nothing about why – which means you learn nothing transferable.

Be honest about what a result means

A significant result is not a fact about the world. It is a statement that, if there were genuinely no difference, data this extreme would be uncommon. Some proportion of significant results are wrong, by construction.

Effects also shrink on retest. The first run catches a favourable fluctuation, and the honest expectation is that a real improvement will be smaller than the test suggested.

None of which is an argument against testing where it is viable. It is an argument against treating a single underpowered test as evidence, and against building a roadmap on top of it.