A/B testing: how to run a test your list can actually answer
Two versions, one change, and a winner picked on what people did. The idea's simple. The trap is a test too small, too short or judged on the wrong number.
- 001A/B Testing
- Comparing two versions of a webpage or email to determine which one performs better based on specific metrics like conversion rates or click-through rates.
- Change
- One thing per test, or you won't know what won
- Judge it on
- Clicks or placed orders, chosen before it starts
- Run it until
- Your list has finished opening and enough people have acted
- Works on
- Subject lines, emails, flows, sign-up forms and site pages
What's worth an A/B test, and what's just rearranging
Test the things that would change what you do next. Those answers keep paying for months.
- The offer on your sign-up form, or whether it needs one
- What the first email in a flow leads with
- How long a flow waits before the reminder
- Whether the last email in a sequence needs a discount at all
- The layout of a product page that gets real traffic
Button colours and emoji in subject lines are different. Whatever difference they make tends to be small, and a small difference needs a huge test to show up. Run one on an ordinary send and you get a coin toss with a confidence score attached.
The number that decides whether a test means anything
It isn't how many people got each version. It's how many did the thing you're measuring. A test sent to thousands can still come down to a few dozen clicks per version, and a gap that small is noise.
Then there's time. Klaviyo lets you set how long a campaign test runs before the winner goes to everyone else. Set it short and the verdict comes from whoever was holding their phone. Let it cover the hours your list actually reads, evenings included.
On a smaller list, stop asking one send to prove anything. Repeat the same idea across several campaigns and compare the pattern. Here's how to size a Klaviyo test before you send it.
Pick the winning metric before the test starts
Not opens. Apple's Mail Privacy Protection loads email in the background, read or not, so an open-based winner partly measures who reads on Apple Mail. Use clicks, or placed orders where you've got the volume.
Write down what you'll do with each result before you launch. If the answer's the same either way, don't run it. And when a test loses overall, cut it by new and returning customers before you bin the idea, because a failed test can hide a version that won with the people your flows had warmed up.
Related terms
FAQs
When should I stop an A/B test and pick a winner?
When you've reached the numbers you sized it for, and the test has covered a full cycle of how your list reads or your site sells. Not when one version pulls ahead early. Early leads come and go, and stopping the moment one looks good is the quickest way to crown a fluke.
Can you A/B test a flow in Klaviyo?
Yes. You can test versions of a flow email against each other, and the split happens as people come through. A flow only sends when somebody triggers it, so results build slowly, often over weeks. Leave it alone until enough people have seen both versions, then pick the winner on clicks or orders.
Is testing three or four versions better than two?
Usually not. Every extra version splits the same audience thinner, so each one gets fewer clicks and the result takes longer to mean anything. Two versions with one clear difference answer more than four that each change a little. Save bigger tests for very large sends or a lot of site traffic.
Rather we explained it on your own account?
Bring your Klaviyo account to a growth consultation. We'll walk through what this means for your numbers, in plain English, and what we'd fix first.
Half an hour with Theo, our founder.
We're all about relationships built on trust, mutual respect and a shared vision for success. If that sounds like your vibe, let's make some waves together 🌊
