It is possible to A/B test your newsletter. For example, do you want to A/B test your open ratio, click ratio or number of unsubscribes? You can! In this article I explain how an A/B test works, how to set it up and how to analyze the results.
Why A/B testing?
How does an A/B test work?
Tips for A/B testing
Setting up an A/B test for a newsletter
Analyze A/B test results
And the winner is...
Why A/B testing?
With A/B testing your newsletter, you test what works best in an email. Is it a blue button, or a green button? Does a clear subject work best or does a teasing subject generate more opens?
How does an A/B test work?
In short, in an A/B test, you create two versions of your newsletter. When you send out the newsletter, the test begins. A sample is taken from the total audience and this sample is split in half. One half receives version A of your newsletter, the second half receives version B. Once the test ends, the winning version is sent to the entire target group.
Before you put an A/B test live, think about what you want to accomplish with the results. This article will tell you what you can do with the results of your A/B test.
Tips for A/B testing
In A/B testing, you can test for three outcomes:
Open ratio
Click ratio
Unsubscribes
We'd like to give you some tips on what you test on:
Open ratio. To test your open ratio, have the sender's name, email subject or pre-header differ from each other. Always test one of the three. Otherwise you will never know which change was the deciding factor in choosing the winner
Click Ratio. We measure the number of clicks in your newsletter. Your click ratio is about the content of your newsletter. Make the content of variant A different from variant B. Change only a button, hero image or text. Which variant gets the most clicks? Test it and make sure your content matches your target group.
Unsubscribes. We measure the number of unsubscribes per newsletter. Many people unsubscribe from the newsletter because the content is not interesting. Try to put yourself in your customer's shoes: why would you unsubscribe from a newsletter? Are there no interesting offers in your newsletter? Or is it the articles themselves? Test thoroughly why your customers unsubscribe from the newsletter.
Always test one element at a time. If you want to test all three indicators, weigh them all equally. So each key indicator has a weight of 1/3. Are you testing two grades at the same time? Then we weight them equally 50/50.
Setting up an A/B test for a newsletter
Step 1. Create a new newsletter
You start by creating your newsletter. In step 1 you select your target group.
In step 2, the magic of A/B testing begins.
Step 2. E-mail settings
Now you are ready to create your A/B test. To enable A/B testing for your newsletter, click the 'A/B testing' slider under A/B testing:
Now you will see that the screen for A/B testing expands. It is now possible to fill in everything for your A/B test. You now have this screen in front of you:
You don't have to worry if your target group is big enough or not for an A/B test to be successful. Our system will tell you this for you:
That way you know where you stand. We recommend a target group of at least 400 profiles. Is your target group smaller? Even then your A/B test will still be carried out. Just keep in mind that your test results will be less representative.
Step 3. Explanation of all fields in your A/B test
The screen you see in front of you can now actually be divided into an A variant (top) , the test settings (middle) and B variant (bottom). I will cover all the fields you see in front of you now.
The A/B test settings
Here you fill in the following fields:
Test duration in hours. Fill in how long you want to test your newsletter. Specify in whole hours. Testing for half an hour is not possible.
Which indicators do you want to test for? Fill in what you want to test for. You can choose from Open Ratio, Click Frequency and Unsubscribe. If you choose all three, we will test the newsletters on all three parameters and you can decide which newsletter will be the winner.
A variant:
The A variant is the first variant you see on the screen. This is where you enter the following fields:
Sender name
Sender e-mail
E-mail subject
Preheader text
Delivery timeframe
Send date
Resend email
Whether to resend unopened emails
This is the same as preparing a newsletter.
B variant:
The B variant is the second variant you see on the screen. Here you fill in the following fields:
Sender name
Sender e-mail
E-mail subject
Preheader text
Have you filled out everything? Then go to the next step.
Step 4. Design
In the next step you will create your newsletter. At the top you will see a dropdown menu where you can choose between variant A and variant B:
Always create variant A first. Variant A is duplicated live to variant B. This way you don't have to create your newsletter twice. Now save version A.
When you have saved Variant A and continue in Variant B, the variants will be disconnected from each other! Then the changes are no longer carried over when you change anything else in variant A.
And continue with variant B. In variant B you make some changes that you want to test. You'll see that variant A stays as it is.
Have you created your newsletter? In the summary, you can see once again that you are running an A/B test:
Keep in mind that your A/B test will not send all your emails.
With large numbers, the A/B test is sent to between 300 and 400 profiles. Even when your target audience contains 100,000 profiles. We have tested that you need between 300 and 400 profiles to determine a significant winner.
Do you send out your newsletter? Then you can see in the overview under 'newsletters' that your newsletter is being tested. You can see this in the status.
Results of your A/B test
You can see the results of your A/B test live. To view the results, click on the three dots behind the newsletter and choose "A/B test results." This is your dashboard for your A/B test results:
On the left-hand side you can see the probability in percentages for the winner: will it be variant A or variant B?
In addition, you have Variant A and B with the performance:
Curious about the 'regular' statistics that you always have in the statistics of a newsletter? Click on one of the lines of variant A or variant B. This will take you to the unique performance of the newsletter concerned.
Please note that during testing, the regular statistics dashboard does not show anything yet. Once the newsletter is sent to the entire target audience, the regular statistics dashboard will flood with data.
And the winner is...
Our software automatically chooses the winner based on the collected data. It depends entirely on your target group how many emails are sent from the winner. This also depends on the testing period (1 hour or 10 hours makes a difference).
Do you find the results convincing enough halfway through the test? Then press "Make this variant the winner". That variant then wins and the test is aborted. The remaining e-mails that were waiting will be automatically converted to the winning variant and sent.
The winning variant is sent when the test is over. Does your A/B test take 12 hours? Then the winning variant is sent after exactly these 12 hours.
💡 Have you run an A/B test in which the results are the same? Then variant A has won 50% and variant B has won 50%. Our system will then pick a random winner, as the statistics are completely equal.
Finally, you can find all your active A/B tests under e-mails > campaigns > A/B testing:
Is your A/B test finished? Then your A/B test is listed under 'all newsletters'. You can see that this has been an A/B test, because it says 'A/B test' in green behind it:
Good luck with your A/B tests! 🙌