How to measure the effectiveness of your email campaign with a control group?

How To Measure The Effectiveness Of Your Email Campaign With A Control Group?

Photo Credit = mwewering@pixabay

With email campaigns, the high level performance metrics like open rates, click thru rates and conversion rates are great. These basic metrics generally provide a good overview of your campaigns.

And, these metrics can be your starting point for more in-depth analyses particularly in campaign optimisation. However, these metrics might not be the best way to draw a conclusion on the effectiveness of your email campaigns.

The concept of a control group is simple. A control group allows marketers to accurately and scientifically measure the effectiveness of the campaign with a baseline.

Here are some important questions to think about before going further…

How much incremental sales uplift did the email campaign alone generate?

Which email campaign generated the most incremental sales uplift if you had multiple campaigns running in the same period?

Did the email campaign change your customer purchase behaviour? And, how?

We have 2 simple examples below to demonstrate how to measure the effectiveness of an email campaign with and without a control group.

#1, Not using a control group

Example 1,

You sent out an email campaign a couple of weeks ago. Now, it is the most exciting time to measure the sales uplift from the campaign by doing some post campaign analysis.

Here are some dummy numbers.

The send size was 10,000. The offer validity was 7 days. 2500 customers purchased within the 7 days. The conversion rate was 25% for this campaign. The average spend per customer was $250.

Here is a table with the numbers.

Email Campaign XYZTarget Group
Offer Validity7 days
Send Size10,000
Customers Purchased2,500
Conversion Rate25%
Average Purchase$250
Sales Uplift$625,000

Boom! This campaign had generated roughly $625,000 sales uplift (excluding other costs associated with the campaign).

Wait! Do you see any problems with this approach?

Now, consider these questions below…

How many of these 2,500 customers would have purchased in the same period anyway?

How much of this $625,000 would have been spent by customers without receiving the campaign?

All we can tell is that X number of customers spent $Y in period Z. And, we assume that… this email campaign alone generated most if not all of $Y.

#2, Using A Control Group

Example 2,

Here are some dummy figures and a chart with a 5% control group.

A 5% control group from 10,000 customers would have been (10,000 X 5%) = 500 customers. These customers were randomly selected from the 10,000 targeted customers (target group).

  • Test Group
  • Control Group

With a 5% control group, 20% of your customers in the control group purchased without receiving the email. They spent an average of $180 per customer.

Email Campaign XYZTest GroupControl Group
Offer Validity7 days7 days
Send Size9,500500
Customers Purchased2,300100
Conversion Rate24%20%
Average Purchase$250$180
Sales Uplift$575,000$18,000

Since the control group (baseline) represents the entire target group, the target group would have spent $360,000 without receiving the campaign. (10,000 X 20% X $180) = $360,000

Again, this would have been the sales generated from the control group regardless of if these customers received the campaign or not.

We know that…

The test group generated (2,300 X $250) = $575,000

The control group generated (100 X $180) = $18,000

Thus, the target group generated ($575,000 + $18,000) = $593,000

The incremental sales uplift excluding other costs from the email campaign was ($593,000 – $360,000) = $233,000 

You can also analyse other activities from each group with questions like…

Did this campaign create a halo effect?

Why did customers in the control group spend?

By understanding the effectiveness of the campaign, you can effectively optimise your next campaign moving forward.

#3, Control Group and Test Group

To effectively measure a campaign, data-driven marketers need to define a control group when preparing for the targeting. And, measuring with a control group is a standard practice when doing post-campaign analysis.

Essentially, a control group is a subset of customers you are targeting with a campaign (target group) but these customers are NOT going to receive the campaign. These customers will be excluded from the campaign to ensure that they do NOT receive the email at all. The idea is to create a reliable baseline to compare the result against with the test group.

A test group is a subset of customers who are going to receive the campaign. So that, you can compare the activities of your control group against with your test group to determine the effectiveness of the campaign.

The composition is shown below.

Control Group = Target Group – Test Group

#4, How to define a control group

How to determine the size of a control group?

It is sufficient to have a 5% to 10% control group in most cases. Generally speaking, the smaller the campaign the bigger the control group.

However, it also depends on other factors such as expected response rate. If customers are less likely to respond to a campaign (re-activation campaign), then a bigger control group is required. If customers are likely to respond to a campaign (an email offer to your high value customers), then a small control group is required. In some cases, you might even need to use a 20% control group to obtain statistically significant results.

How to select a control group?

The customers in the control group need to be selected randomly from the entire target group. Alternatively, customers can be selected with every N customer in the target group (Pick every 3 customer from the list). So that, it effectively represents the entire target group by NOT simply selecting the top N customers.

There are a few ways to select random customers depending on tools that you are most familiar with. There is a RAND() function in Excel which generates random numbers. There are other ways to use SQL, SAS and R to generate random numbers depending on the size of your customer base.

This post provides an overview of using a control group for email campaigns. However. there are other ways to apply the concept to different campaigns such as creating stratified control groups across multiple segments within a target group.

Connect with us to discuss more on your post-campaign analysis or if you have anything to add to this post.

Related Posts
Are You Getting The Most Out Of Your Customer Data?