Your Klaviyo AB test sample size was never big enough to crown that winner

The green banner is a verdict on two hours, not on your list

The banner says variation B won, and a confidence score sits next to it. That score is the part that convinces people, because a number makes it look like maths happened. Behind it, a tenth of the list was split three ways, a couple of hours passed, and the platform picked a side.

Then the winning subject line went out to everybody else, and somebody wrote always lead with urgency in a Slack message that'll be quoted in a strategy deck six months from now. A green banner isn't evidence, it's a timer running out. The verdict happened quickly, and quickly isn't the same as right.

Count the opens per variant before you trust a winner

Take a 5,000 person list, because it's a completely normal size and the numbers only get worse below it. A test group of a tenth of the list is 500 people, and split across three subject lines that's roughly 167 recipients per variant. If one in four of them opens, your winner is being crowned by somewhere between 35 and 45 opens.

That's barely a sample. It's a focus group you'd never trust for any other decision in the business. A gap of three or four points between two subject lines sits comfortably inside the noise at that size, and the losing line could easily be the better one on a bigger send. Klaviyo's confidence score is honestly calculating win probability from the data you gave it, and that data was never enough to ask the question.

So before the next test, convert your test group into raw opens or clicks per variant and run that number through a proper significance calculator. A dramatic swing might be genuinely detectable on your list. The small difference most subject line tests actually produce needs a sample far beyond what a rushed split can give, and knowing which case you're in is a bit more discipline before you hit send.

The test closes before your list has finished opening

Sample size isn't the only problem, and the next one does more damage. Klaviyo's default test duration is a matter of hours, not because that's statistically sound but because the person running the test wants an answer before lunch. Your list doesn't open email on a schedule that respects the timer.

Work openers open in the first hour. Evening scrollers and people who check email once a day open twelve, twenty-four, sometimes forty-eight hours later. Close the window at hour two and the verdict was built by the most reactive slice of your list, then applied to everybody who behaves nothing like them.

Pull the time-to-open curve from your last five or six campaigns, reading opens over time instead of the final total. If half your opens land after hour six, a two hour test was never going to catch a representative picture. It caught whoever was already holding their phone.

The fix costs almost nothing. Set the duration to where your own curve flattens, which for a lot of DTC lists means twelve to twenty-four hours instead of two, and let the winner go out the next morning. You lose a few hours of send timing on one campaign. You get back a verdict that includes the people who were asleep when it went out.

Judge it on clicks or revenue, because Apple opens email for you

Even a perfectly sized, perfectly timed test hands you a false winner if you judge it on opens. Apple's Mail Privacy Protection has been pre-fetching and auto-opening email for a meaningful share of iPhone inboxes since 2021, and those recorded opens say nothing about which subject line was better. A machine fired them.

You'd never crown a landing page test on a metric both variants can hit passively, and the same logic applies inside the inbox. So switch the winning metric to clicks, or on a campaign to placed order rate, which Klaviyo lets you set in the test itself. On a flow it only offers opens or clicks, so somebody reads the placed orders by hand. It just isn't the default, so most accounts never touch it.

What to check before you let the dashboard decide

Five things, run through before Klaviyo runs the numbers for you.

• Raw sends per variant once the list is actually split, not the headline test group share
• Whether that number, through a proper significance calculator, could detect a lift you would act on
• Where your time-to-open curve flattens, and whether the test duration is set to match it
• Whether the winning metric is opens, clicks or placed orders, given how much of the list reads on Apple Mail
• Whether the gap between variants clears the margin of error, not just the other variant's number

If a test fails more than one of those, treat the result as a hunch worth trying again, not a decision worth building a strategy on. The check takes ten minutes, and it decides what the banner is allowed to mean.

Fewer tests on bigger splits beat more tests on slivers

Look, the instinct after a shaky result is to run more tests, faster, on smaller slices, and that makes it worse. The boring option fixes it. Run fewer subject line tests, with two variants instead of three because every extra variant shrinks the usable sample, sized to detect a difference that would genuinely change what you send next.

For smaller lists where a proper test group still won't clear a useful sample, and plenty won't, stop treating each send as its own verdict. Run the same subject line structure across several campaigns and compare cumulative performance instead. That builds a sample from real volume over time, instead of letting one rushed afternoon decide your tone of voice for the quarter.

Klaviyo isn't lying to you. It's reporting exactly what happened to 167 people in the first two hours, and you're the one who decided that counts as the truth about the other 4,833. The accounts behind the results we publish run fewer tests than you'd expect, and every one of them was sized before it was sent.