How to set up an A/B Test in beehiiv

Open rate is one of the most important metrics for any publisher or creator. After all, high open rates mean more eyeballs on your content and more ad dollars in your pocket. A/B testing helps newsletters improve open rates and ensure the best subject line is being sent to readers.

If you’re familiar with A/B testing but want to run a better experiment, review these six considerations for a better A/B test. Otherwise, read on for a 101 on A/B testing in beehiiv.

What is an A/B Test?

In order to send the best subject line to readers, publishers will often create multiple subject line versions to test which one produces the highest opens. A/B testing can be done with two or more versions of subject lines.

These test subject lines are sent to a smaller (but still representative) sample of the email list, chosen at random. The sample group is then split evenly, with each group receiving one version of the subject line. The test runs for a set period of time (typically a few hours), and then the open rates from the different subject lines are analyzed. The subject line with the highest statistically significant open rate is sent to the remainder of the audience.

For example:

Let’s say we have an email list with 100,000 subscribers and we want to run an A/B test with two subject lines announcing a new feature. The test will go to 20% of our subscribers over a 3 hour test period.

• Subject Line A: “Check out this amazing new feature”

• Subject Line B: “We just released A/B testing, see the details”

The test is launched at 8am, and runs for 3 hours. During these 3 hours, the test email will be sent to 20K subscribers, which is 20% of the total audience. 10K subscribers will receive Subject Line A, and 10K subscribers will receive Subject Line B.

After 3 hours pass, we will check to see if the test achieved statistical significance and which subject line performed better. If statistical significance is not achieved, we will simply see which subject line led to a higher number of opens. This winning subject line will be sent to the remaining 80K subscribers.

How to Set Up Your A/B Test in beehiiv

You can also watch a video on how this works here: https://youtu.be/7Ph7HeBA5NE

Note: A/B testing is available on our Grow plan or above. You can learn more about our plans here.

To set up a test within a new post, navigate to Configure —> Email Settings.

Add in the first version of your subject line that you want to test, and then click into Create A/B Test. A field for Subject Line B will appear.

Note: You can test two subject lines on the Grow plan or up to four on the Scale plan.

From here, you can click into A/B test settings to see more details. (You must be on the Scale plan to modify these settings).

You will see two fields: Duration and Sample Size.

Duration: This is how long your test will run for. The duration defaults to 180 minutes, but you can choose any length of time between 5-240 minutes. We will run the test for this duration to determine the best subject line. Once the test is complete, we will send the winning subject line to the remainder of your audience.

Note: Once you set up your duration and start your test, you are not able to end your test early.

Example: If you set the duration to 180 minutes, we will send the test versions of your subject line to the subscribers in your sample size. After 180 minutes, we will determine which subject line performed best. That winning subject line will be sent to all of your other subscribers at that time.

Sample Size: This is how many subscribers will be included in your test. We default to 20%, but you can choose any size between 1-100%. Based upon the percent you choose, we will automatically show you how many users fall into that test group. We will then equally split that full test group into smaller groups that will each receive one version of your subject line.

Example: You have 100K subscribers total, and you set the sample size to 20%. We will run the A/B subject line test with 20K subscribers. If you have 2 versions of the subject line, we will send each version to 10K users. After your test duration is complete, the remaining 80K subscribers will receive the winning subject line.

Another way to A/B Test in beehiiv

Sometimes, instead of sending different test versions of your subject line to a small subset of your audience, you want to test with your entire list of subscribers. You can do this in beehiiv too.

All you have to do is set your Sample Size equal to 100%. This means the test versions of your subject lines will be sent to all your subscribers. You can set up two or more versions of subject lines, and your audience will be split evenly amongst the different versions.

There are pros and cons to subject line testing with your full audience.

Pros: Since you are using your full audience, you have a higher chance of seeing statistically significant results.

The email is also sent to your entire audience at the same time. If you send your emails out at specific times, timing might be important to your brand and newsletter. In this case, sending to all your subscribers once might work better.

Cons: When structured this way, the non-winning versions of your subject line are sent to a large portion of your audience. When you test with a small sample size, you can find the best subject line and send it to your remaining audience. When sending a test to your full audience though, if you have two versions of your subject line, half of your audience will receive a subject line that will end up being the losing version.

Regardless of how you set up your test, it's important to make testing a regular and frequent practice for the success of your newsletter. A/B testing is a process of iteration, not perfection. Test, test test!

We hope this was helpful! Again, remember to review the six factors for better A/B tests once you understand the basics.

Happy testing!