Improving Email Marketing Results With Pre-Testing

As Internet marketers, we love our testing, and one of the greatest benefits of email marketing over direct marketing is the immediacy of testing results.  By ‘pre-testing’, we can use that immediacy to improve the performance of our email campaigns.

A typical A/B test usually involves developing two different versions of an email (e.g. different subject lines, including personalization, etc.), splitting the list of subscribers into two randomly selected groups, and sending a different version to each group.  The test is often run multiple times, results are analyzed, and the information is used to inform future campaigns. 

Most marketers start with what I’ll call ‘macro tests’ which involve larger issues such as testing different layouts, best time of day and day of week to send, etc.  All of these type of macro tests are very important and establish best practices and guidelines for an email program.

However, there are situations in which elements specific to a campaign need to be tested – I’ll refer to those as ‘micro tests’.  For example, maybe the creative director and product manager disagree on which photo should be used in the email as a hero shot or there are questions about the arrangement of words in the subject line (i.e. which are most important to place toward the front).  You could just A/B test the two approaches, sending each version to 1/2 of the list.  However, if one version significantly outperforms the other, then you would have lost opportunity by sending out the worst performing version to 50% of your list. 

Let’s look at the results (similar to one of our client’s recent campaigns) of an email that was A/B tested with 200,000 subscribers and in which version A outperformed version B:

Typical A/B Test Scenario

Typical A/B Test Scenario

The good news is that we did 20% better than if we would have sent version B to the entire list. However, the bad news is that we performed 20% worse than if we had sent version A to the entire list.  Of course, we didn’t know which would be the best version prior to the send.  Pre-testing allows us to reduce the risk associated with sending a worse-performing email to a large percentage of our list.

A pre-tests involves deploying the initial A/B test to a smaller, but statistically significant percentage of subscribers first and then sending the ‘winning’ version to the remainder of the list.   For example, using the same number of subscribers and response rates in the example above, a pre-test sent to 20% of the list would generate the following results:

Pre-Testing Scenario

Pre-Testing Scenario

In this example, pre-testing improved results by 16% over straight A/B testing.  The greater the performance between the two versions, the more benefit provided (and risk-reduced) by pre-testing.

A few caveats about pre-testing:

  • Pre-tests are not suitable for all situations.  For example, there are some tests (like testing a new enewsletter layout) that you are going to want to run multiple times involving as many subscribers in the the sample as possible.  Also, you need to allow at least 24 hours between the pre-test and the send to the reaminder of the list so that you have enough data to reach a conclusion, so if the email is time sensitive, you may not have time for the pre-test.
  • Even though you want the pre-test groups to be small, the groups need to be large enough to be statistically significant. (for more on sample sizes and statistical relevance, read Wayde Nelson’s response in a MarketingProf knowledge exchange answer)
  • To help validate your approach to pre-testing, run a few tests where you conduct a pre-test with your two versions and then deploy an A/B test to the remaining subscribers.  If you don’t see the same results between your pre-test and full A/B tests, then you need to pre-test with a larger sample size or check to see if something else is impacting results (e.g. day of send).

This entry was posted on Monday, April 20th, 2009 at 1:29 pm and is filed under Email Marketing, Metrics, Online Strategy. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.

9 Responses to “Improving Email Marketing Results With Pre-Testing”

  1. OnlineBusinessOpportunities Says:

    Great! Just continue your work!

  2. горячий шоколад для похудения Says:

    Будто похудеть при помощи шоколада?

    Сладкоежки, желащие сбросить лишний вес, горячий шоколад для похудения понимают, насколько сильной становится тяготение к лакомствам, когда ей оглашается сражение.
    и все же имеется возвожность баловать себе сладостным и одновременно освобождаться с
    жировых отложенией – шоколадка про похудения ChocolateSlim снимает возражение меж, на первый взгляд, несовместимыми вещами.

    Чтоб выписать Шоколад про заполните форму и нажмите кнопку “Выписать”.

    С вами свяжется свой клерк для уточнения условий доставки

    Состав и полезные свойства

    Какао.Главным компонентом любого шоколада считается
    шоколадное дерево. Нежели
    его более, другими словами, чем чернее шоколадка, и
    слава богу. В какао держатся препарата, коие положительно влияют возьми работу сердечно-сосудистой порядка и нормализуют влияние.

    недавно шпанские эксперты исследовали, касающееся капиталом самочувствия молодых людей
    с 12 по 17, коие придерживались различных диет и водили здоровый стиль жизни.
    В результате оказалось, будто у
    ценителей шоколада численность жирных отложений
    имелось существенно ниже, чем у остальных.
    Разъясняется таковым, будто шоколад убыстряет метаболизм.

    Даже несмотря на высокую калорийность, он не дает способности
    жиру продавиться в организме в качестве отложений.
    И еще одно увлекательное надзор.

  3. Lorie Says:

    Con la cantidad de buenos profesionales que hay en este país, la mayoría de photocall están iluminados en vez
    de alumbrados.

  4. Christen Says:

    El trabajo de Biddle se caracteriza por una alta saturación de colores, la combinación de formas y
    un estilo que se aproxima a reconocidas obras de pintores señalados,
    entre ellos Piccaso y Dalí.

  5. Says:

    Poquito a poco la réflex se fue quedando en casa y llegó un momento en que me
    decidí a apostar por las cámaras sin espejo incluso para trabajos profesionales
    debido a su ligereza, fiabilidad y buena calidad de imagen.

  6. Anke Says:

    Las imágenes de Nelson Marques representan la elegancia,
    el dinamismo y la maestría de un saber hacer aprendido de los más grandes fotógrafos y de la experiencia.

  7. fotografos de bodas madrid Says:

    Considerado un profesor de la foto en blanco y negro, Peter Lindbergh destaca por sus imágenes cinematográficas, que
    han influenciado el planeta de la moda por su realismo y emoción.

  8. nombres de fotografos famosos - Top diez Fotógrafos Internacionales Karen Copito Says:

    [...] along with you wish to receive guidance concerning fotografos de bodas madrid baratos famosos 2015 [Relacionados Página Web] kindly go to the webpage. Madrid son algunos ejemplos), aparte de, como es natural, en toda la [...]

  9. fotografo de bodas sevilla Says:

    Este neoyorquino ha influido en las tendencias del mundo de la moda durante más de 20 años, colaborando en campañas para firmas como Prada,
    Dolce & Gabbana, Valentino y Versace.

Leave a Reply: