Measuring Conversion Lift with a Facebook Holdout Test

Have you ever wondered how your retargeting campaigns are actually impacting sales? If customers didn’t see or click on your ads, would these individuals be buying your products anyway?

Facebook Experiments offers an easy-to-use feature, called holdout tests, for testing your hypotheses about conversion lift. In a nutshell, a holdout test will withhold ads from a subset of your audience, allowing you to compare conversion performance vs. those who did see ads. You can then measure the likelihood that ads led to conversions.

Setting Up a Holdout Test

To set up a holdout test, make your way to the Experiments section of Business Manager. Look for the beaker icon in the menu.

Next, choose the ad account you want to work in. Look for the Holdout Test option, and select “Get Started.”

Now, you’ll see a page with the options to set up the details of your test.

First, you can choose to test account-wide, for a campaign group, or for individual campaigns. Note that if you have campaigns with potentially overlapping audiences, you’ll want to include all those campaigns for an accurate test.

Next, choose the start and end date. Note that you need to obtain at least 100 conversions for each conversion event you’re tracking in order for results to show up, so choose a time period that will allow for enough conversion volume.

Finally, choose up to 10 conversion event(s) for inclusion in the test. Note that you can select from Facebook Pixel events, app events, or offline events.

Think about the volume each individual event gets (to reach the minimum number of 100), and also think about representing various stages of the funnel. For ecommerce, include events such as Add to Cart in addition to Purchase. For lead gen, think about including events for asset downloads, as well as sales inquiries.

On the next page, you’ll see an option to review test details before launching the test.

Reporting on Test Results

Once you reach at least 100 conversions for any one event being tracked, you’ll start seeing data for your test. Facebook’s report will also tell you the percentage of confidence in the conversion lift seen. They recommend at least 90% confidence for reliability.

In the results section, you can see data for all of the events you’re tracking. You can also select individual events to see more details, as well as a campaign breakdown, if you selected individual campaigns.

Note that you can extend a test before it reaches the end date, if you see that you’re not getting a significant enough pool of data.

Start Testing!

Ready to start testing? Suggest a holdout test as a new option for vetting out campaign performance. The results can either make a proof point for the ads you’re running or show where you might need to change course.

Have you tested this experiment type in Facebook Ads? We’d love to hear about your results if so!