For a successful A/B test, you first need to have a hypothesis. A hypothesis aims to solve a specific (conversion) problem. This is how you make the test measurable. Therefore, in an A/B test, you test a hypothesis and not an idea.
How to set up a hypothesis for my A/B test?
To set up a hypothesis, you can use Craig Sullivan's hypothesis kit. To do this, you fill in the following:
Because I've noticed that [data and/or feedback from previous research]
I expect that [the change/improvement you want to test] will produce [impact you expect] and
I will measure that impact using [the data measuring tool you will be using].
Fill in the blanks and your test idea has changed to a hypothesis. This ultimately makes it easier to properly read your results from your A/B test.
When your A/B test is over and you have a winner, the remaining profiles that haven't received an email yet will receive the winning variant. After that your A/B test is finished.
Analyze the A/B test results
Once the test duration of your A/B test has ended, Reloadify automatically determines a winner based on the statistics. The remaining profiles from the target group then receive the winning variant of the newsletter. Time to analyze the results!
Analysis is about much more than just seeing which variant won or lost. It depends entirely on your hypothesis. What did you want to test and how was it achieved? It could well be that your "losing" variant actually becomes a winning variant if you use a different segment. So think carefully about which segment you give which A/B test.
In fact, there are conclusions to be drawn even from the losing variant, just as there are from the winning variant. Why did the winning variant win? And might the losing variant have fewer dropouts than the winning variant? Become wiser from your test results. Also from individual variant A and individual variant B.
How to save the results of your A/B tests
Keep the results of your A/B tests well. Without a good test archive, all insights are lost. Besides, if you don't save everything properly, chances are you will run the same tests.
The following elements are indispensable in your report of A/B tests:
Screenshots of your A/B test results
Which customer won/lost the test
Insights the analysis brought you
Over the years, you will thank yourself for keeping this archive.