What do I do with the results of my A/B test?

Before you begin an A/B test, you must first have a hypothesis. A hypothesis is measurable and aims to solve a specific (conversion) problem. Therefore, in an A/B test you are testing a hypothesis and not an idea.

You can use Craig Sullivan's Hypotheses Kit for this purpose:

Fill in the blanks and your test idea has changed to a hypothesis. This ultimately makes it easier to properly read your results from your A/B test.

When your A/B test is over and you have a winner, the remaining profiles that haven't received an email yet will receive the winning variant. After that your A/B test is finished.

But then what?

This depends entirely on your hypothesis. What did you want to test and how was it achieved? It could well be that your "losing" variant actually becomes a winning variant if you use a different segment. So think carefully about which segment you give which A/B test. Indeed, even from the losing variant conclusions can be drawn, just like from the winning variant. Why did the winning variant win? And did the losing variant perhaps have fewer unsubscribes than the winning variant? Become wiser from your test results. Also from the individual variant A and individual variant B.

Chances are that your losing variant is actually the winner in some segments, information that you can benefit from now and in the future. Analysis is about much more than just looking at which variant won or lost. Also look at the individual segment data for example.

Suppose you do your first test tomorrow. Will you remember in a few years exactly what the results were and why? Probably not. That's why you should save the results of your A/B tests. Without a good test archive, all insights are lost. In addition, there is a good chance that you will run the same tests if you don't keep everything well.

Basically, you can set up your archive however you like, as long as it contains the following elements:

Over the years, you will thank yourself for keeping this archive.