As Internet marketers, we love our testing, and one of the greatest benefits of email marketing over direct marketing is the immediacy of testing results.  By ‘pre-testing’, we can use that immediacy to improve the performance of our email campaigns.

A typical A/B test usually involves developing two different versions of an email (e.g. different subject lines, including personalization, etc.), splitting the list of subscribers into two randomly selected groups, and sending a different version to each group.  The test is often run multiple times, results are analyzed, and the information is used to inform future campaigns. 

Most marketers start with what I’ll call ‘macro tests’ which involve larger issues such as testing different layouts, best time of day and day of week to send, etc.  All of these type of macro tests are very important and establish best practices and guidelines for an email program.

However, there are situations in which elements specific to a campaign need to be tested – I’ll refer to those as ‘micro tests’.  For example, maybe the creative director and product manager disagree on which photo should be used in the email as a hero shot or there are questions about the arrangement of words in the subject line (i.e. which are most important to place toward the front).  You could just A/B test the two approaches, sending each version to 1/2 of the list.  However, if one version significantly outperforms the other, then you would have lost opportunity by sending out the worst performing version to 50% of your list. 

Let’s look at the results (similar to one of our client’s recent campaigns) of an email that was A/B tested with 200,000 subscribers and in which version A outperformed version B:

Typical A/B Test Scenario

The good news is that we did 20% better than if we would have sent version B to the entire list. However, the bad news is that we performed 20% worse than if we had sent version A to the entire list.  Of course, we didn’t know which would be the best version prior to the send.  Pre-testing allows us to reduce the risk associated with sending a worse-performing email to a large percentage of our list.

A pre-tests involves deploying the initial A/B test to a smaller, but statistically significant percentage of subscribers first and then sending the ‘winning’ version to the remainder of the list.   For example, using the same number of subscribers and response rates in the example above, a pre-test sent to 20% of the list would generate the following results:

Pre-Testing Scenario

In this example, pre-testing improved results by 16% over straight A/B testing.  The greater the performance between the two versions, the more benefit provided (and risk-reduced) by pre-testing.

A few caveats about pre-testing:

• Pre-tests are not suitable for all situations.  For example, there are some tests (like testing a new enewsletter layout) that you are going to want to run multiple times involving as many subscribers in the the sample as possible.  Also, you need to allow at least 24 hours between the pre-test and the send to the reaminder of the list so that you have enough data to reach a conclusion, so if the email is time sensitive, you may not have time for the pre-test.
• Even though you want the pre-test groups to be small, the groups need to be large enough to be statistically significant. (for more on sample sizes and statistical relevance, read Wayde Nelson’s response in a MarketingProf knowledge exchange answer)
• To help validate your approach to pre-testing, run a few tests where you conduct a pre-test with your two versions and then deploy an A/B test to the remaining subscribers.  If you don’t see the same results between your pre-test and full A/B tests, then you need to pre-test with a larger sample size or check to see if something else is impacting results (e.g. day of send).

It’s amazing how often I hear the same misconceptions and misunderstandings from prospects we talk to, clients we work with, or others when discussing Internet marketing.  While businesses have become more savvy with respect to the web, some still have much to learn. 

The following are five of the most frequent Internet marketing myths that I come across:

It’s really about the latest and greatest trend.  While technology continues to amaze, some things don’t change.  Content is still king and in order to engage you have to provide something of value.  The printing press probably wouldn’t have had such an impact initially if Gutenberg had decided to print a book of mutton stew recipes instead of the Bible.  In the end, technology can’t overcome bad design, poor strategy, inferior quality, or terrible customer service. 

I can handle email subscribers in the same was as my direct mail list.  Direct mailers sometimes have a difficult time understanding the repercussions of emailing to subscribers too frequently or to those who didn’t provide clear permission.  Junk mail delivered by the United States Post Office, at worst, ends-up in the recycling bin, while unwanted email triggers spam complaints which in turn tarnishes your email reputation and makes it more difficult to get your communications to those who really want it.  Just because it only costs fractions of a penny to send an email to an email address doesn’t mean that you should.

I’ve implemented XYZ analytics application so I have done all that I need.  Implementing a service like Omniture Site Catalyst or Google Analytics is a great start, but it is just a start.  Companies need to ensure that they have trained staff dedicated to analyzing the information provided by tools like Site Catalyst.  These applications have many wonderful reports but the ‘canned’ reports won’t tell you everything in you need (often they only provide a small portion), and you have to have skilled people focused on analyzing the data in order to gain the insights needed to improve results.

SEO is free – I can just have someone already on staff handle it.  There are companies that handle search engine optimization in-house and are very successful.  But those companies will be the first to acknowledge that it’s not free.  SEO is still such an art and science that you need to dedicate real resources to truly understand how to succeed.  Also, because the rules are continually evolving, a considerable effort needs to be made to stay current and that takes time and commitment.  Companies who think they can buy a book and task someone without search marketing experience to handle SEO in her ‘spare time’ soon discover that approach is a recipe for failure.

I control the conversation.  Companies can foster, facilitate, participate, and even impact the conversation but they can’t control what is being said about their organization.  Customers are having conversations about organizations and their products and services on Facebook, Twitter, You Tube, blogs, Amazon (product reviews), etc.  There is nothing that a company do to stop it and attempts to only backfire.  Organizations must stop worrying about control and learn how to participate.

What Internet marketing myths would you add?