Use A/B Tests to improve your communication

Thomas Edison Quote

There is no better way to learn and understand, than trial and error. But to make a trial and error approach worth it, there must be a way to quantify or recognize which aspects of our trials bring us towards failure and which ones bring us towards success. This is why AltosMail has the “A/B Test” function; an extremely powerful tool.

Different products and different campaigns can demonstrate what a niche of customers finds more interesting, but it’s very important to go beyond this. Any business, in any field, must find out exactly which type of communication style or way of presenting their products has more appeal on their customer base, in order to increase the level of interest and interaction.

One of the best ways of going about this is to use A/B Tests in Mail Marketing Campaigns, so let’s find out exactly how.

A/B Tests allow a business to create two or more versions of a single email which can be linked and sent without repetitions. Each of the variants should have only one, or at the most two changes between each email, so that any behavioral differences performed by the leads, can be easily linked to a specific cause.

This is similar to what professional motorsports racing teams do, when setting up the vehicle for a new race track. If too many changes are made at once, the pilot will not be able to tell which ones actually improved the handling, and which ones were counterproductive. To avoid this, only one or two changes will be made each time the pilot goes out to test the vehicle, and when he comes back, the engineers will evaluate with him if those changes should be taken further, cancelled, and if new ones should be added on, thus avoiding confusion and in the long run saving a lot of time and money.

In mail marketing, the newsletters and the customer base can be compared to the pilot and the vehicle, while the engineers are represented by the marketing staff, which uses the feedback from the A/B Test to figure out which variables improved the performance and how to keep going in that direction with future newsletters. As we mentioned, the changes between each variant should concern only one aspect of the email, and can be things such as different subjects, images, buttons, or promotions.

At this point it’s probably best to look at one example of an A/B Test. This particular test was used to find out which type of subject achieved a higher opening rate. Since getting leads to actually open and read marketing emails is the first step towards a successful mail marketing campaign, this is certainly one of the first A/B Tests that any business should perform.

AB Test Subjects

The screen shot shows the two different subjects that were given to a single newsletter, and how these two variants were divided amongst the lead list (50/50). The important thing to notice is how the two subjects aim to appeal two very different types of customers:
A)When mentioning the arrival of new products, this subject tries to appeal those looking for new items that might be trending.
B)When mentioning the arrival of new products, this subject tries to appeal those looking for a nice discount or a good deal, rather than something new.

The two categories of customers that the subjects address in this case are very distinct, but it could be very important to understand if the customer base is mostly made up of “innovators” looking for new/trendy items, rather than “scavengers” looking for a good bargain and low prices.

In this particular example the actual newsletter should hold a more generic content, because it would have to appeal to both types of leads and invite them to click through to the website in any case, however keeping in mind that the actual goal of this A/B Test was not to see how many customers would land on the website, but rather how many leads would open the email based on how appealing the subject would be to them.

It’s therefore also very important to set the right goal when creating the A/B Test, so that all the data, analysis and reports generated by AltosMail will be aimed at recording the opening rate, rather than other behavior such as the CTR. This can be easily set during the creation of the A/B Test, thanks to a scroll menu which allows for the setting of this “winning criteria” between the two versions:

Winning Criteria

Once the newsletter is sent, all that is left to do is check the reports with all the analysis to see which version performed best. AltosMail provides a series of reports automatically which can be very helpful for the marketing staff in trying to define the customer base and how to communicate to it. Here are some:

Test AB Statistics 1 AB Test Bounced AB Test Funneling AB Test Engagement

The results from this particular test would have shown that the customer base is more reactive to new products and trendy items, meaning that it’s mostly made up of “innovators” or “early majority”, which could be of great help when setting up future campaigns and promotions, providing a very important stepping stone towards higher sales and an increased revenue.

This was a fairly basic example, but to understand the full potential of a tool like this, let’s take things one step further and consider a more complex A/B Test that a skilled mail marketing team could conduct.

Now that the best type of subject has been figured out, it would be important to figure out how to get the most clicks (CTR) in the email, thanks to an attractive and efficient CTA (Call To Action). The skilled mail marketing team might want to find out which is the best sentence to place next to the button that the leads should click, inviting them to visit the landing page. Not just that, but the marketing team also wants to find out if the best way to present this (graphically) is by putting the sentence above, below, on the right or on the left of the button. These would therefore be the variables:

  • Sentence 1 = “Click here”
  • Sentence 2 = “Go now”
  • CA = Caption Above the button
  • CB = Caption Below the button
  • CL = Caption Left the button
  • CR = Caption Right the button
These 6 variables would provide 8 different versions of the email, as follows:

A) CA + Sentence 1
B) CA + Sentence 2
C) CB + Sentence 1
D) CB + Sentence 2
E) CL + Sentence 1
F) CL + Sentence 2
G) CR + Sentence 1
H) CR + Sentence 2


Once the eight versions (A to H) have been created, the marketing team has to decide how many leads should receive each one, as a percentage of the total lead list used for this test. The distribution might look something like this:

A) 15%
B) 15%
C) 10%
D) 10%
E) 15%
F) 15%
G) 10%
H) 10%


This is a very thorough test which AltosMail can allow and perform, even if it’s suggested for a more expert marketing team, the possibility is there!

Performing regular A/B Tests to figure out important aspects of a customer base is a vital tool that can help any business figure out the strategic and effective ways to approach and communicate to them.

AltosMail is a perfect tool for this, so if you have never performed an A/B Test, it’s probably time to start with us!

Share on FacebookShare on Google+Tweet about this on TwitterShare on LinkedIn