All About Testing

Contents
Why you should test - the benefits of email testing

What you should test - choosing variables & metrics to test

Who you should test - segmenting your list for testing

When you should test - time and frequency of sending

How you should test - methodology and tools for testing

10

Summary and Conclusion

23

Introduction
Never stop testing and your marketing communications will never stop
improving. Testing results reveal important facts about your potential
marketing communications success.
Senders want their messages to be delivered and read, and recipients
want relevant content in manageable quantities - free of spam, of course.
Testing helps bring these two goals together by identifying the most
valuable content in your message and the optimal sending strategy.
Before delving into the specifics of why you should test, here are some
notes on how this white paper is organized.
Icons. This white paper uses the following icons to help you find
important information:
WARNING! E.g. There is no silver bullet for sending perfectly
timed email campaigns. Dont assume any general benchmarks
- you must test your own subscriber base to figure out which times work
best.
GM-TIP: E.g. your subscriber base will most appreciate your
email campaigns when you send timely and relevant messages.
Testing your campaigns is all about optimizing how relevant your
message is and finding the best time to send it.
HowItWorks: These icons explain IT jargon and processes in
laymans terms.

back to top

Why you should test - the benefits of

email testing
GraphicMail helps you promote business offers and keeps members
up to date and informed. It allows for engagement via email, text
messaging/SMS and social media all while being able to fully customize
communications to appeal to different individuals at different times.
Test to keep your email list clean and active. Dont waste money on hard
bounces or likely-to-remain unopened sends; find your most engaged list
segment and optimize your communication efforts with them.
Test to get to know your subscribers. Gather knowledge about your target
audience and why they respond the way they do to your campaigns. You
may be able to link different responses to target audience characteristics,
such as demographics, interests, and more.
Test to get ahead of your competitors. Only 31% of digital markets test
regularly, while 41% dont test or test infrequently (see Adestra
Econsultancy study by the Email Marketing Industry Consensus). There is
also a direct correlation between those who test regularly and those who
see higher ROI on email campaigns.
Test to improve your overall campaign efforts. Generate valuable insight
into your campaign effectiveness and eliminate future guesswork for
campaign strategies. Generate the content that is most likely to be
opened and acted upon by your recipients.
Test to make sure your content is so relevant, it gets forwarded - thus
increasing your subscriber base.

back to top

What you should test - choosing the

variables and metrics to test
It may seem daunting to try set up your own thorough plan for testing,
but here is a step-by-step breakdown of the process, beginning with an
overview of important email testing metrics.
Delivery rate: The process of measuring delivery rates by format, ISP or
other factors and delivery failures (bounces, invalid address, server and
other errors). An inexact science.
Open rate: The number of HTML message recipients who opened your
email, usually as a percentage of the total number of emails sent. The
open rate is considered a key metric for judging an email campaigns
success, but it has several problems. The rate indicates only the number
of emails opened from the total amount sent, not just those that were
actually delivered. Opens also cant be calculated on text emails. Also,
some email clients only scan message content without actually opening
the message, which is falsely calculated as an open.
Click-through rate (CTR): Total number of clicks on email link(s) divided
by the number of emails sent. Includes multiple clicks by a unique
user. Some email broadcast vendors or tracking programs define CTR
differently.
Bounce rate: In email marketing, the bounce rate relates to the delivery
of your campaign. It tells you what percentage of addresses didnt
receive your message because it was returned by a recipient mail server.
High bounce rates damage your reputation and delivery process.

back to top

WARNING! ISPs will shut down IP addresses sending large

volumes of bounced emails. Soft bounces should be monitored,
but hard bounces (e.g. bad mailboxes) should be immediately flagged
and removed from future mailings.
Forward (or share) rate: The process in which email recipients send your
message to people they know, either because they think their friends will
be interested in your message or because you offer incentives to forward
messages. Forwarding can be done through the recipients own email
client or by giving the recipient a link to click, which brings up a
registration page at your site, in which you ask the forwarded to give his/
her name and email address, the name/email address of the person they
want to send to and (optionally) a brief email message explaining the
reason for the forward. You can supply the wording or allow the forward
to write his/her own message. AKA viral marketing.
Unsubscribe rate: To remove oneself from an email list, either via an
emailed command to the list server or by filling in a Web form.
Spam complaint rate: The frequency of spam complaints you receive as a
sender.
Once you are familiar with these testing metrics, you can decide what to
test.

back to top

Step 1: State your business objective. What do you seek to

accomplish with your email campaign? For example, is the purpose to
publish a newsletter or to use email marketing to drive traffic to your
website; to generate awareness or inform customers; to increase sales/
conversions/ROI, strengthen company brand, cost savings, customer
retention, lead generation, demonstrate authority in your field etc?
Step 2: Know your baseline. Identify your control message - what
you have been using in the past - to compare your test results against. If
you dont have any good existing content or are just getting started, you
can simply make two versions of whatever youd like to say to your
subscribers and test which one works better.
Step 3: Structure your test to meet your goal. Choose the right
variables and measurement strategies that are most likely to have a big
impact on your objective. Think about the variables and metrics which
are most important to achieving your goal.
Here are some examples if you need some help:
A professional organization is seeking to increase awareness of an
event and pump up attendance numbers: focus on subscriptions,
referrals, click through on event links (which topics are most
interesting?), conversions (e.g. RSVPs)

back to top

 A sales representative or consultant wants to demonstrate expertise

to select customers - focus on click-throughs, whitepaper downloads,
conversions
A start-up wants to gather information on potential customers for
life cycle marketing: gather actionable intelligence into subscriber
preferences & behavior, e.g. which opt-in sources produce most
engaged subscribers, how active subscribers behave based on the age
of their account
A marketing department wants to improve upon past campaigns:
benchmarking against standard metrics from past campaigns, such as
delivery rates, open/CTR, conversions
A restaurant wants to increase business on its dead days - track
customer location visits (feedback from survey), coupon redemptions
of customers (whos likely to use them) - segment and attract these to
come on dead days with incentives in email message.
If you still arent sure what to test, the MarketingSherpa 2012 email
marketing benchmark report found the following to be most valuable to
test for increased campaign effectiveness:
1.

Target audience

2.

Landing page

3.

Subject line

4.

Call to action

5.

Personalization

The above-mentioned points can easily make for a page each, in order to
fully explain, but are worth keeping in mind.

back to top

Who you should test - segmenting your

list for testing
Remember, one of the most important rules in email marketing is to set
and meet expectations with your subscribers - deliver what you say you
will when you say youll deliver it. While testing may alter the flow of your
regular sending routine, it is a good idea to test more drastic changes on
newer subscribers.
Consider segmenting your email lists based on the following
characteristics:
Frequency/engagement preferences - how often do your subscribers wish
to engage with you? Remember, you can always ask for correspondence
by way of a subscription form. You have the tools to segment your email
lists into those who have: forwarded, commented on, shared socially or
clicked on a link. This gives you a clear indication of whos talking and
those who arent interacting.
Domain address - find out what domains are most common in your
address list, and sign up for an email account with those major senders.
Send your email campaign to yourself at those addresses as part of your
testing routine. You can also use GraphicMails Inbox Preview to see how
your emails will render in different inboxes.
GM Tip: Even if you dont have any other data beyond email
address, you can segment your subscribers by those who have
opened vs those who never have, subscribers who have recently engaged
vs those who havent.

back to top

When you should test - timing and

frequency of sending
The time you send your emails can tell you when your subscribers are
most likely to engage with your CTAs and at what frequency they prefer
to engage.
1. Timing
Any test sends should be run at the same time to account for any
variations in testing, e.g. testing a variation today and another tomorrow
may skew the results based on the different response likelihood of
each day of the week. Instead, you need to send to users seeing
different variations at the same time. CITE Example: http://www.
marketingexperiments.com/blog/research-topics/email-marketing/timingand-email-marketing-test.html
There is no silver bullet! Dont paint your entire audience with the same
brush. People differ and you need to keep that in mind.
GM-TIP: Segment your lists based on user type, e.g. personal
(name@hotmail) vs professional (name@companyname.com). As
a general rule of thumb, you can expect personal emails to respond more
readily to emails sent during commuting hours, lunch breaks, weekends,
etc. whereas professional emails are likely to be active during normal
working hours.

back to top

2. Frequency
Different subscribers have different preferences for the amount of email
theyd like to receive. Make sure you ask your subscribers for their
feedback on frequency preferences - and be sure to use this data!
a. Consider asking your subscribers in the subscription form how often
they would like to receive newsletters from you (e.g. daily, weekly,
bi-monthly, monthly).
b. It is also a good idea to occasionally check in with your subscribers
about their preferences for your email mailings. You can even add a
quick survey link to your newsletter sends to get their responses.
c. When subscribers choose to unsubscribe, be sure to give them the
option to give feedback for why they are unsubscribing (e.g. too much
volume, not relevant). If you see a rise in your unsubscribe rate and
the majority of responses point to high volume, you may want to
consider cutting back your newsletter sends.

back to top

How you should test - methodology and

tools
Creating varying content to test:
Once you have decided upon your objective and how to best measure it,
you must generate different versions of content to see which is most
effective. Here are some examples of various ways to test the following
parts of your email message:
1. Subject line
Subject lines serve to identify the message as being on topic for what
your subscribers signed up for - usually this includes some sort of call to
action to draw the reader in. The first line of text should complement the
subject line and continue to draw the reader into opening the email and
ultimately clicking your CTA.
Offer lead, e.g. Order ___ now (insert new product name)
Editorial lead, e.g. Read our new ___ here (insert headline from article)
Personalized, e.g. Hey ___, would you (insert call-out with topic of
their interest)
2. From line / sender address
The From line is the first contact point for how your audience identifies
you as the sender. Although the From line is typically a company name,
it can be a variation or a persons name. Testing between the two - a
company name or an individuals name - on different types of messages
may show a considerable difference in response rates.
E.g. Andy Warren from andy@mybusiness.net vs. GraphicMail from
newsletter@graphicmail.com

10

back to top

WARNING! Dont stray too far from your branding here,

otherwise there is a chance the subscriber wont recognize who
the sender is and hit the spam or delete key.
WARNING! There is a recent trend toward straying away from
using no-reply email addresses, as subscribers can become
frustrated with an inability to contact the sender should they require any
additional information or assistance.
3. Header

The header gets your audience to keep reading. It is the pitch of the
email, along with the first few lines of text. This is also known as snippet
text.
4. Images
Images are important in supporting text and adding visual diversity.
Its important to keep in mind the volume of images with relation to
text both should be balanced; and as important in making sure your
demographic can relate to the images.
GM Tip: Consider running an A/B test on image variations to see
what works best for your subscribers; you can use segmentation
to test how various age groups, demographics, etc will respond to
different images.

11

back to top

5. Call to Action (CTA) / Offer

Test the CTA
CTA Wording: the goal is to convince someone why they should click
further. E.g. Get your free whitepaper, Start saving today,
Get instant access,.
GM Tip: If your subscriber base isnt as familiar with you, they
may be scared off by offers which seem too good to be true or
which demand commitment. Try the wording See plans and pricing
rather than Free Trial or Sign up for a free trial to see if it helps
improve click through.
Here are a few of things to consider when trying to up your CTR
(click through rate):
CTA images
CTA location in the email
CTA format, e.g. html button graphic vs plain text link
Offer testing:
Hard sell (Purchase now $20.00) vs soft sell
Pricing/discount level (20% vs 30%)
Tack on free bonus offer vs no bonus offer (e.g. free shipping)
One product vs. multiple products
Target segment, e.g. frequent vs. lapsed customersmaybe on
segment values free shipping while the other wants $5 off next order

12

back to top

6. Links and landing pages (for CTA). Yes the second most important
variable to test is the landing page, have a look at the benchmark report
mentioned earlier. If a landing page does not reinforce the email, then
visitors will drop off, i.e. poor conversion rates. Also keep in mind the
ergonomics of mobile technology. Screen size, usability and the like
especially when placing links inside text.
GM-TIP: test the difference between specific landing pages and
more general landing pages. Depending on fluidity of your site
navigation, visitors may convert at much higher rates based on where
you send them.
WARNING! If the same URL is used for multiple links within your
message, be sure to keep the links separate with different
wording so you can distinguish which ones work best (e.g. which link
placement is more effective for CTR).
7.Dynamic content: Personalization and target audience Segment your
lists by location/age/gender/etc, add their name to subject line, and
include information about recent purchases and behaviour/interests.
8. The Look & Feel: Always keep in mind the demographic for which you
are designing material. While content is key, aesthetics matter as much
and no one is going to suffer through a poorly formatted or unattractive
newsletter for very long.
Look at what is trending in terms of design, but also remember to
keep things simple. Too many font changes, for instance, makes your
communications look amateurish and stick to a minimal color palate.

13

back to top

WARNING! This is more of a long term testing area; be patient

and dont do anything rash.
First, you should ensure that your email is viewable in the most used
email client of your subscriber base; look at your opens by email client to
check, e.g. desktop vs. webmail vs mobile. Some email marketing service
providers offer Inbox Preview testing tools that make it easy to see how
your newsletter looks in various email clients and on mobile screens.
Testing the look and feel of your emails
Layout: template used
Style: message type promotional, informational, procedural, relational
Colors and fonts - dont stray too far from your brand image
Mail format - text vs column vs all vertical
Length of message - long with all info included, or short with clickable
links to see more
Here is a graph from a study by Experian Marketing Services Email
Marketing Study, Acquisition and engagement tactics (December 2012)
What type of testing do you perform on email campaigns?
TEST

PERCENT

Subject line

97%

Creative

81%

Frequency

39%

Time of day

50%

Day of week

36%

HTML vs. text

11%

Call to action

43%

product placement

21%

number of products

28%

friendly from

11%

multivariate

11%

14

back to top

Spam checking
Check the spam level of email content. Run your client through spam
filter tools, if available; have a look at these common words and phrases
used in spam messaging. While this list is by no means exhaustive, the
following are the most well-known.
Official spam words
100% FREE

SPAM

Click here
Risk free
No fees
Take a look at this article for a more comprehensive breakdown of spam
key words: http://www.bloomtools.com/articles/spam-trigger-words-toavoid.html
GM TIP: Be sure to be especially careful with these words in the
subject line. It is a good idea to test small changes in subject line
wording to find what works best, which sometimes means what seems
the least spam-like to your subscribers.
Preview your email
If automated preview tools are unavailable, sign up for common email
test accounts (after checking opens by email clients in your subscriber
base) and send test emails to those accounts.
It is also important to test cross screen testing, such as mobile devices
and tablets.

15

back to top

Being able to test your newsletters before they get to your subscribers
allows your material to hit the mark first time and guarantees a much
higher click through rate. If youre a more advanced user you can
even fix some HTML bugs at this point yourself, although by no means
necessary you can drag and drop edit your newsletter!
HowItWorks: Ever wonder how ESPs track all those emails
to obtain that rich data? To supply link tracking, open HTML
reporting, bounce back handling, and other features, ESPs place links
within emails (e.g. attached to images) which send a message to the
ESPs server when an email-client fetches data to load the email. This
data retrieval is used to produce reporting on when and how users click
through your emails.
A/B Split testing
A/B split testing helps you know what works and what doesnt. When you
have hard results, its easier to make on-going campaign strategy
decisions which can be more effective from the outset.
HowItWorks: A/B split testing sends two or more different
versions (e.g. email A vs email B) to random splits of your
recipients. The ESP tracks results to show how your newsletters perform
and what factors are affecting their performance. Most commonly, A/B
testing is done on either subject line or newsletter content.
GM-TIP: Mix it up! Test various content over time to optimize
different parts of your email message. GraphicMail split testing
allows you to create up to five different mailers, each with their own
subject line or with different content. Then divide them into group A and
group B for sending. Our system tells you which one worked best and
then sends that newsletter to everyone. Easy as that. Saves you time,

16

back to top

money and guarantees a higher open and conversion rate.

Example: Foolproof AB split test plan STEP 1: Split your list into three parts:
a. Recipients of test A 10% of your list
b. Recipients of test B- 10% of your list
c. Recipients of the winning email rest (80%) of your list
STEP 2: Track the following 3 rates:
a. Open Rate
b. Click Through Rate
c. UnSubscribe Rate
Question to keep in mind when running split tests
1. What are you focusing on to prove and improve?
2. What are the assumptions as to what will and will not work?
3. Are the changes big enough, can they be bigger?
4. Do the proposed tests have high potential for seriously improving
results?
5. How will you measure the outcomes, is this a valid measurement?
6. Will your tests answer the hypothesis you set?
7. Can you re-use the results in later campaigns?

17

back to top

Multivariate testing (A quick note)

A/B testing is the most common form of testing used in marketing today,
relying solely on measure the effects from changing values of only one
factor.
HowItWorks: A/B testing demonstrates one level of one
factor vs the control, but this process does not measure effect
of interaction across different factors. Combinations of different factors
may produce additional effects when used at the same time. Multivariate
testing examines changes when all factors are changed together and all
combinations accounted for. However, this increases the complexity of the
test and can decrease its usefulness if the sample size is not sufficiently
large to test across the many combinations.
Deliverability testing:
While there are numerous testing options marketers have to choose from,
the most important aspect often goes amiss getting the email
delivered. Making sure your email hit their mark isnt as simple as doing
an A/B split test; marketers have to fine-tune various aspects and with
every change, monitor its deliverability through different domains like
Yahoo, AOL and Gmail.

18

back to top

GM-TIP: When you use an ESP like GraphicMail to deliver your

emails, we take care of automated IP switching and domain
delivery throttling. This means that should an ISP ever choose to block
the emails from your IP address, our sending servers will throttle the
delivery flow (i.e. restrict to X thousand emails per hour) of messages
and switch sending to another IP address, which should resolve the
blocking.
GM-TIP: [List other GM testing tools: http://www.graphicmail.
com/site/features_power_intelligence.aspx]

Best practices for email campaign

testing
Test only the bare minimum or test only whats relevant to your
objectives!
If you have a large mailing list, consider testing a significant portion to
yield a winning result and then send the remaining portion the winning
message.
Frequency of sending: regular sending schedules are more
appreciated than sporadic sending; but this depends on the expectations
you set when your recipients first subscribed. Too often will frustrate
recipients, while too little will lose their interest. Find the balance over
time. It also depends on your industry - for example, if youre a news
agency or event company, you might send more frequently than say a car
dealership, and even more than a physiotherapist.

19

back to top

Test your inactive subscribers liberally, but test your highly engaged
customers conservatively to avoid upsetting them or failing to meet their
expectations for delivery.
WARNING! Never stop testing. When done consistently,
split testing can help improve the bottom line substantially.
Additionally, you can see subscriber trends over time which may not be
apparent from a single campaign test.

20

back to top

Conclusion
Remember, you are not just testing to meet your primary business
objective. You are also testing to improve the quality of your email
sending skills. You must use these powers to elevate the value of your
message content in respect for the privilege youve been granted to send
to a subscribers inbox.

21

back to top