What do I do with the results of my A/B test?

For a successful A/B test, you first need to have a hypothesis. A hypothesis aims to solve a specific (conversion) problem. This is how you make the test measurable. Therefore, in an A/B test, you test a hypothesis and not an idea.

To set up a hypothesis, you can use Craig Sullivan's hypothesis kit. To do this, you fill in the following:

Fill in the blanks and your test idea has changed to a hypothesis. This ultimately makes it easier to properly read your results from your A/B test.

When your A/B test is over and you have a winner, the remaining profiles that haven't received an email yet will receive the winning variant. After that your A/B test is finished.

Once the test duration of your A/B test has ended, Reloadify automatically determines a winner based on the statistics. The remaining profiles from the target group then receive the winning variant of the newsletter. Time to analyze the results!

Analysis is about much more than just seeing which variant won or lost. It depends entirely on your hypothesis. What did you want to test and how was it achieved? It could well be that your "losing" variant actually becomes a winning variant if you use a different segment. So think carefully about which segment you give which A/B test.

In fact, there are conclusions to be drawn even from the losing variant, just as there are from the winning variant. Why did the winning variant win? And might the losing variant have fewer dropouts than the winning variant? Become wiser from your test results. Also from individual variant A and individual variant B.

Keep the results of your A/B tests well. Without a good test archive, all insights are lost. Besides, if you don't save everything properly, chances are you will run the same tests.

The following elements are indispensable in your report of A/B tests:

Over the years, you will thank yourself for keeping this archive.