Holdout Testing: A Guide

Hi Friends,

It’s been a busy week. Let me catch you up!

I spent last weekend in the Pocono Mountains at an Airbnb; nothing short of magical.

Airbnb has gotten quite the negative rap on social media lately, but here are two tips to ensure you don’t get messed up with a cuckoo host:

1. Only choose Superhosts. It mitigates the biggest risk—having a psychopathic host. 😂

2. Rigorously read both positive and, more importantly, negative reviews. Are the potential negatives things you can live with?

That’s your bi-monthly travel tip. 😏

Now, onto the BEEF.

When joining Jones Road from OLIPOP, working with Cody (our CMO) was an adjustment to how we did things at Oli. If you read last week’s newsie, OLIPOP was omnichannel and had a much larger focus on splashy brand stuff, whereas JRB is much more tempered and measured.

The biggest question Cody always asks is: “How are we measuring this?”—and not just on splashy store openings or gifting campaigns but also on SMS campaigns and Direct Mail, etc.

Anything that has a variable cost to get to the masses gets measured.

I’ve learned more about holdout testing in the last year than I knew was possible, and we’ve had some really strong wins lately.

I will educate you on everything you need to know about the subject and end with a real-life example and win.

Come learn with me.

• What: The Art of Holdout Testing

• How: How to Conduct a Holdout Test Like a Pro

• Case Study: A Real-Life Holdout Testing Success Story

This newsie is brought to you by Tapcart.

CX and retention are two sides of the same coin. As a CX or Retention operator, your job is to align them to work in harmony to ensure your customers aren’t just satisfied—they're coming back for doubles.

From my experience, though, customer retention today is either over-glamorized or misunderstood.

It’s either “everything needs to be perfect” with no clear tactic, or it’s just “send as many emails and texts as you can.”

Most brand founders don’t get Retention.

I recently sat down with Tapcart to create a chapter of a new FREE resource they are launching this week.

Say hi to The Retention Rundown.

I tackled all these topics:

• The intersection of retention and CX: Why retention is the heart of CX

• Retention beyond communication channels: It's more than just email and SMS

• The rise of retention experts: Why brands are investing in dedicated roles

• JRB’s approach to retention that drives delightful customer experience and fuels growth

• BFCM and retention: Leveraging retention strategies during the holiday season

• Maximize your customers’ LTV with retention marketing

I can’t wait to hear how you like it. Check it out here:

What: The Art of Holdout Testing

Before I go deep into explaining how to set up a basic holdout test, let’s take a step back and explain what this even is.

I’ll be honest, a lot of this math/measurement stuff used to fly right over my head, so let me do this:

As a marketer, your job is to ascertain if the magic you are creating is actually impacting more than if you did absolutely none of it.

Given the variables in a business that change daily as well as shifts in the economy generally, the only way to know if something is working is to take a group of people that are (as close to) exactly alike and try your marketing spiel on 80% (the treatment group) while “holding out” the other 20% (the holdout group) and tracking any incremental lift from the treatment group.

How: How to Conduct a Holdout Test Like a Pro

In an effort to keep this simple, let’s continue chatting about your fantastic lemonade stand. As an example, let’s use Direct Mail.

Step 1: Define Your Goal:

First things first, you need to have a crystal-clear goal for your holdout test. What do you want to measure? Is it the effectiveness of a new marketing channel, the impact of a specific campaign, or the magic of a brand-new lemonade recipe?

Whatever it is, write it down and keep it front and center throughout the whole process.

For the purposes of our example, let's stick to the classic direct mail approach. Your goal is to see if sending out fancy postcards to potential customers leads to more lemonade sales.

Step 2: Choose Your Audience:

Remember the "normal group" and the "fancy group" we talked about earlier? Well, it's time to select those lucky lemonade enthusiasts who will be part of your holdout test.

In this case, you'll want to take a big group of similar customers (a cohort) who have similar tastes and preferences.

Divide them into two groups randomly – one that will receive the fancy postcards (the treatment group) and the other that won't receive anything (the control group).

Step 3: Set Your Baseline:

Before you launch your holdout test, you need to know where you're starting from. Measure the current lemonade sales of both groups (treatment and control) without any fancy postcards in the mix. This is your baseline, the starting point to measure any changes from.

Step 4: Test and Track

Here comes the fun part! Time to unleash your fancy postcards on the treatment group while the control group enjoys their regular lemonade. Let this go on for a certain period of time, like a week or a month – whatever makes sense for your business.

Step 5: Crunch the Numbers:

After the test period, gather all the data you've collected and put on your detective hat! Calculate the lemonade sales for both groups, treatment, and control. Now, here's where the real magic happens – compare the results.

In the case of direct mail, if you can not match back your customers’ usage using their address or email via Shopify, you can also use a coupon code on the card, which will help you know who came from the card and who did not.

Did the fancy postcards make a splash and lead to more lemonade sales in the treatment group compared to the control group? If you see a significant difference in sales between the two groups, you've got yourself a winner! The fancy postcards are doing their magic.

Step 6: Draw Conclusions:

Now that you have the results, it's time to draw some reasonable conclusions. Did the fancy postcards bring in enough additional sales to make them worth the investment? Does the increase in lemonade sales justify the cost of printing and mailing those postcards?

This is the moment of truth, where you decide whether to go all-in with the fancy postcards or maybe tweak your marketing mix based on the results.

Case Study: A Real-Life Holdout Testing Success Story

At JRB, we've been using SMS marketing to keep our customers in the loop about new product launches, exclusive offers, and makeup tips. We believed that these SMS messages were driving repeat purchases and boosting customer engagement.

But like all smart marketers, we wondered, "Can we take our SMS game to the next level? Is there a way to measure the true impact of our SMS marketing efforts?"

To find out, we worked with the Postscript team to run a holdout test using three different groups of our loyal customers:

Group 1: The Control Group (20%)

This was the holdout group – the 20% of customers who would not receive any SMS messages at all during the test period. They would continue enjoying their Jones Road Beauty products without any SMS interactions. Everything else would stay the same.

Group 2: The Standard SMS Group (40%)

The next 40% of customers would receive the usual SMS messages, just like they always had. This was our baseline, the group we could compare other results to.

Group 3: The Increased Send Group (40%)

The final 40% of customers got special treatment! They would receive SMS messages more frequently, with additional product updates and offers from Jones Road Beauty.

We set our holdout test in motion. We sent SMS messages to the Standard SMS Group as usual and increased the frequency of SMS for the Increased Send Group.

When we analyzed the data, the results were truly glamorous.

We observed a significant increase in revenue per recipient between the standard send group and the holdout group, and again between the standard and the increased send group.

We were able to quantify that, in this case, sending texts to our customers visibly increases the revenue we see from them, and sending at a slightly higher frequency increases revenue per recipient even more.

Armed with these valuable insights, we can confidently take the next step, focusing our efforts on the Increased send strategy.

We found the perfect balance—the sweet spot where SMS marketing made the biggest difference in our customer relationships.

The important caveat is that if we keep increasing, we might see an elevated unsubscribe rate and a decrease in long-term engagement due to customer fatigue or irritation, but we will keep testing.

I’ll be sure to let you know how it goes.

That’s it for this week!

Any topics you'd like to see me cover in the future?

Just shoot me a DM or an email!

See you next week,

Eli 💛

P.S. I’ve got an open sponsorship slot that just opened in August. If you are interested in learning more, just reply to this email or shoot me a DM on Twitter (X app).