{"id":69,"date":"2009-04-20T13:29:48","date_gmt":"2009-04-20T19:29:48","guid":{"rendered":"http:\/\/www.shaydigital.com\/?p=69"},"modified":"2009-04-21T10:03:20","modified_gmt":"2009-04-21T16:03:20","slug":"improving-email-marketing-results-with-pre-testing","status":"publish","type":"post","link":"http:\/\/www.shaydigital.com\/index.php\/insights\/improving-email-marketing-results-with-pre-testing\/","title":{"rendered":"Improving Email Marketing Results With Pre-Testing"},"content":{"rendered":"

As Internet marketers, we love\u00c2\u00a0our testing, and\u00c2\u00a0one of the\u00c2\u00a0greatest benefits of email marketing over direct marketing is the immediacy of testing results.\u00c2\u00a0 By ‘pre-testing’, we can use that immediacy to improve\u00c2\u00a0the performance of our email campaigns.<\/p>\n

A typical A\/B test usually involves\u00c2\u00a0developing\u00c2\u00a0two different versions of an email (e.g. different subject lines, including personalization, etc.), splitting the list of subscribers\u00c2\u00a0into two randomly selected groups, and sending a different version to each group.\u00c2\u00a0 The test is often run multiple times, results are analyzed, and the information is used to inform future campaigns.\u00c2\u00a0<\/p>\n

Most marketers start with what I’ll call ‘macro tests’ which involve larger issues such as testing\u00c2\u00a0different layouts,\u00c2\u00a0best time of day and day of week to send,\u00c2\u00a0etc.\u00c2\u00a0 All of these type of macro tests are very important and establish best practices and guidelines for an email program.<\/p>\n

However, there are situations in which elements specific to a campaign need to be tested – I’ll refer to those as ‘micro tests’.\u00c2\u00a0 For example, maybe the creative director and product manager\u00c2\u00a0disagree on which photo should be used in the email as a hero shot or\u00c2\u00a0there are questions about the arrangement of words in\u00c2\u00a0the subject line (i.e. which are most important to place toward the front).\u00c2\u00a0 You could just A\/B test the two approaches, sending each version to 1\/2 of the list.\u00c2\u00a0 However, if one version\u00c2\u00a0significantly outperforms the other, then you would have lost opportunity by sending out the\u00c2\u00a0worst performing\u00c2\u00a0version to 50% of your list.\u00c2\u00a0<\/p>\n

Let’s look at\u00c2\u00a0the results\u00c2\u00a0(similar to\u00c2\u00a0one\u00c2\u00a0of our client’s recent campaigns) of an email that was A\/B tested with\u00c2\u00a0200,000 subscribers and in which version A\u00c2\u00a0outperformed\u00c2\u00a0version B:<\/p>\n

\"Typical

Typical A\/B Test Scenario<\/p><\/div>\n

The good news is that we did 20% better than if we would have sent version B to the entire list. However, the bad news is that we\u00c2\u00a0performed 20% worse than if we had sent\u00c2\u00a0version\u00c2\u00a0A to the entire\u00c2\u00a0list.\u00c2\u00a0 Of course, we didn’t know which would be the best version prior to the send.\u00c2\u00a0 Pre-testing allows us to reduce the risk associated with sending a worse-performing email to a large percentage of our list.<\/p>\n

A\u00c2\u00a0pre-tests involves deploying\u00c2\u00a0the initial\u00c2\u00a0A\/B test to a smaller, but statistically significant percentage of\u00c2\u00a0subscribers first and then\u00c2\u00a0sending the ‘winning’ version to the remainder of\u00c2\u00a0the list.\u00c2\u00a0\u00c2\u00a0 For example, using the same number of subscribers and response rates in the example above, a pre-test sent to 20% of the list would generate the following results:<\/p>\n

\"Pre-Testing

Pre-Testing Scenario<\/p><\/div>\n

In this example, pre-testing improved results by 16% over straight A\/B testing.\u00c2\u00a0 The greater the\u00c2\u00a0performance between the two versions, the more benefit provided (and risk-reduced) by pre-testing.<\/p>\n

A\u00c2\u00a0few\u00c2\u00a0caveats about pre-testing:<\/p>\n