I Did 1,000+ A/B Tests. This Is What Really Matters.

I’ve sent over 100,000,000 emails and completed over 1,000 A/B tests – and you’re about to learn what really matters, and why you should start A/B testing now – even if you’re intimidated.

Don’t worry. I’ll make it super simple so you can get started right away.

My name is Michael Comeau, and I’m a NYC-based Marketing Consultant.

I’ve done test after test in industries like finance, crypto currencies, mental health, and photography to figure out what drives email performance.

Hint: the 80/20 rule applies, big time.

I’ll also show you how to not get fooled by a test, and give you 4 A/B test suggestions you can use right now.

First things first.

What Is an A/B test?

An A/B test (also known as a split test) is when you test two emails against each other to see which performs better. You can test just about anything in your email – the subject line, call-to-action, copy, images, the day or time you send it…you get the picture.

Subject line tests are the most common. That’s when you pit two subject lines against each other to see which one generates more email opens.

Say I was sending you this article in a newsletter.

You set up your email to send two emails with these subject lines:

• Email A: 1,000+ Split Tests Taught Me This

• Email B: Why You Have to Start Split Testing Now

You decide how to run it. In some cases, each email goes to half of your list. In others, the test is performed on a subset of your email contacts. For example, you send email A to 10% of your list and email B to another 10%. And once the test is over, you send the email with the higher open rate to the rest of your list.

If Email A has a 38% open rate and Email B has a 31% open rate, Email A wins.

Simple, right?

To ensure your A/B test is meaningful, test just one element at a time. So in a subject line split test, everything aside from the subject line must be identical: the sender name, the sending date and time, the email content, etc.

If you test more than one element at once, you won’t know which one’s driving performance.

beehiiv makes A/B testing super easy in its Grow and Scale plans. Read this tutorial so you know how to get started.

How the 80/20 Rule Applies to Email Performance

The 80/20 rule, also known as the Pareto Principle, states that 80% of outcomes come from 20% of causes.

This applies to email.

80% of an email’s performance is driven by the subject line.

If a person doesn’t open your email, they can’t read what you have to say.

They can’t click a link to your website.

They can’t listen to your podcast.

They can’t buy your product.

This is why you should split test subject lines starting now. There’s no point in writing a killer 500-word email newsletter if no one opens it.

But more opens isn’t the only reason to start A/B testing.

Understanding Your Email List as a Living, Breathing Organism

Your email list is a beautiful and unique snowflake.

And you must know what your readers care about.

Say you run an email newsletter about playing guitar, and you test these subject lines against each other:

• Email A: How to Shred Like Eddie Van Halenvs.

• Email B: What happened on stage last night…

You’ll find out if your audience wants to learn to play like their heroes (Email A), or if they want to hear stories (Email B).

Assume, of course, the email content was the same - like you were talking about how you nailed a Van Halen solo at a bar gig.

By testing, you take your ego and assumptions out of the mix. You let the numbers do the talking.

Why People Open Emails

There’s only one reason to open an email: to see what’s inside.

So driving email opens is about generating curiosity in the reader’s mind. And it’s a specific type of curiosity – the “what’s in it for me?” kind.

Your job is to write a subject line that implies some value within the email.

That value can come in the form of:

• Information

• Entertainment

• Infotainment - the best of both worlds

So when crafting your email subject lines, ask yourself “How can I let people know they get value by opening my email?

You A/B test emails to find better ways of demonstrating your value to your audience.

So let’s start with one of the most important rules of A/B testing:

Test Big Things or Nothing at All

Do not test minor differences in email subject lines.

If your newsletter is about personal branding, you will see no real difference between subject lines like:

• Email A: My 5 Best Personal Branding Tipsvs.

• Email B: My Top 5 Personal Branding Tips

Why?

Because the subject lines are too close. People won’t be able to tell them apart. You’ll get a bigger bang if you do something like this:

• Email A: My 5 Best Personal Branding Tipsvs.

• Email B: How to Explode Your Personal Brand in 5 Easy Steps

Which one would win?

I don’t know. That’s why we test!

So let’s get you testing.

4 Subject Line A/B Tests You Can Run Right Now

The first thing you should test is your newsletter name against a single selling point.

If you ran a crypto newsletter, this could look something like:

• Email A: Nino’s Crypto Secrets - February 2, 2023 Editions.

• Email B: Why Ethereum Is About to Die

And since lower-case subject lines are so in style, you might as well test those too.

You could test these in a psychology newsletter:

• Email A: The Hidden Psychology of Romantic Attachment

vs.

• Email B: The hidden psychology of romantic attachment

You should also test subject line lengths. Assume you have a newsletter about independent film productions.

You could try:

• Email A: How to Make a Your First Filmvs.

• Email B: How to Make Your First Film, Even If You Have Zero Budget and No Experience

And finally, I’ll give you my favorite test: the “what it is test.”

This is when you test “what it is” against an exciting benefit.

Say we have a video game newsletter focused on Call of Duty.

We could test:

• Email A: The 5 Best Sniper Rifles in Call of Duty: Warzone 2vs.

• Email B: This rifle will 10X your headshot game

In each of these four cases, you might assume you know which subject line will win.

(I’m 95% sure Email B would win across the board.)

The beauty of testing is you find out just how good your instincts are. And the more you test, the more surprises you find.

Your audience might respond to hype and sizzle.

And maybe it won’t.

Testing is the only way to find out. Want more tips and best practices on A/B testing your subject lines? Check this out.

Before we close out, you need to learn when to trust your test results.

This brings us to an important rule.

Do Not Trust a Single Test

Let’s say you run a weekly newsletter about YouTube growth, and your subject line is always something like “John’s Weekly YouTube Tips - January 20, 2023.”

But you get inspired to A/B test it.

So next week, you test these subject lies:

• Email A: John’s Weekly YouTube Tips - January 27, 2023vs.

• Email B: I found the perfect thumbnail formula

Email A gets a 35% open rate. And Email B smashes it with a 43% open rate.

Is Email B better?

Maybe.

Changes to your sending practices can cause a short-term spike in opens because of the novelty factor. All of a sudden, people see your name with an unfamiliar subject line format, and it gets their attention.

There is a risk that your open rate will revert to normal. Complete at least four split tests before you commit to a new subject line practice.

This brings us to another question.

What Is a Meaningful Difference in an A/B Test?

For most email lists, I consider a five percentage point increase in open rate, sustained over four split tests, to be meaningful.

So if your new subject line style keeps on beating your old style, you are on the right track.

Just keep in mind that if you have a small list (under 250 leads), you will see wild divergences in performance.

These are rough numbers, and there are a million factors that impact open rates, including list size, industry, and email frequency.

Keep testing to get those open rates as high as possible.

Just make sure you…

Don’t Base Your Entire Email Strategy on Jacking Up Your Open Rate

All things being equal, having more people read your emails is a good thing.

But don’t fall into the trap of boosting your open rate with over-the-top sensationalist headlines.

You will lose trust with your audience, which will destroy your email engagement.

If you make an exciting subject line, you better deliver on it.

Final Words

There is almost no downside to A/B testing your emails and endless upside.

You don’t need to be a math genius.

If you know that 40% is a higher number than 35%, you’re smart enough to do this.

And over time, you’ll have a better feel for what your audience wants, so you can send them better emails.

Now get testing!

Thank you for reading.

Click here to learn more about Michael Comeau’s email marketing strategies, and sign up to try them out on beehiiv.