AM Solutions

Direct Mail Testing Guide

by Erik Koenig

David Ogilvy, the Father of Advertising, once said, “Never stop testing, and your advertising will never stop improving.”

Testing to optimize the effectiveness of your marketing efforts is a no-brainer. Yet, with so many elements to investigate, testing strategies to consider, and methods to measure performance, knowing where to begin can be a brainteaser. With the right guidance, you can build a sustainable testing program to discover your best audience, learn what motivates them, and refine your campaigns. The right strategy will allow you to maximize your testing budget and your program profitability. Here is how to get started.

Determining What to Test

Direct mail testing does not have to be complicated to be effective. However, it should be calculated and systematic. This takes planning and commitment.

Start by devising your testing ground rules. How often will you test? At what volumes? What is an appropriate test versus control ratio? How often will you test incrementally versus swinging for the fences? (Hint: you should do both.) Each of these depends on the maturity of the channel and how control performance is achieving the KPIs. Set these parameters and stick to them.

From there, identify the order of importance (potential impact) of your testing categories: list, offer, creative, and digital integration are the primary variables in direct mail and are typically tested in that order.

Take inventory of what has worked in the past, as well as what failed. Review competitive samples through a tool like Competiscan or Mintel’s Comperemedia.

Then, use this information to develop a testing roadmap to outline and prioritize test opportunities within each category, complete with a test rationale, hypothesis, and performance projection.

Generally speaking, your mailing list(s) has the biggest impact on campaign performance (optimization and scale), therefore, it should be prioritized and account for the majority of your testing. Testing new data sources, new modeling approaches, different customer segmentation, and more are sure fire ways to move the needle.

Also examine how you can integrate additional direct marketing channels, such as digital or CTV, with your direct mail to boost campaign performance. Scrutinize the creative package for enhancement opportunities, paying close attention to the offer, CTA, format, images, colors, and fonts. It is also important to consider the timing and cadence of your prospecting and retargeting campaigns, such as frequency, seasonality, and delivery day. All these variables can impact response rate, conversion rate, purchase value, customer lifetime value, and overall program ROI.

Testing Strategies for Direct Mail

With your testing roadmap in place, it is time to devise your direct mail testing strategy (design). The two most common designs are A/B and multivariate.

In A/B testing, you are testing two versions of one element (e.g., creative), with one single variance between them, such as the offer or personalization. Although an affordable testing option, A/B can be a timely approach to determining the best performing combination of list, offer and creative. This testing strategy works best for mature campaigns where the current performance is acceptable.

Multivariate direct mail testing allows you to test multiple components of your program at a time. This strategy accelerates the process but may also dilute the benefit, as tests are often conducted in small sets due to available budgets. New mailers who are working to quickly identify a control or those looking to achieve a step-change improvement benefit most from this strategy — provided they can afford the lofty price tag.

Recognizing a gap in the two approaches, some agencies offer proprietary testing strategies that aim to deliver multivariate test results but at a fraction of the cost. By isolating test elements and leveraging indexing, direct marketers can identify the variables that have the greatest relative impact on campaign performance to accurately identify a control combination — whether or not that particular combination was tested. This strategy is attractive to brands who are new to the channel, and to those who are looking to improve the performance of existing programs – even if they have more modest budgets.

Measuring your Direct Mail Test Results

The end goal of a direct mail test is knowledge, not profit. Regardless of the outcome, there are no wasted results. You will learn what works and what does not — and you should never have to spend precious budget to test non-performing data sets or combinations again.

Your overall test objective (i.e., lower cost-per-acquisition or improve ROAS) should not only drive which tests you prioritize, but also determine how you will measure and analyze your results. Rather than obsess over response rates or sales rates, we recommend you focus on striking the perfect balance between cost-per-acquisition and lifetime value. After all, one should inform the other.

To easily track your investment, certainly include directly attributable elements on your mail piece, such as a unique URL or promo code. But don’t stop there, be sure you pair those results with a matchback analysis comparing your sales file to the mail file for a comprehensive view of campaign performance and attribution reporting.

Take caution against making snap judgments based on directly attributable performance alone. Allow ample time to perform your matchback analysis. Due to its longevity, a direct mail read period often lasts 60 to 90 days. Performing a backtest to validate initial test winners can help bring statistical validity to your results. Finally, let the data inform your future campaign strategy, despite what your gut says. Log the results on your test roadmap – and keep testing!

Adopting a Results Mindset

Consumer behavior and preferences are ever evolving, and the direct mail industry is no different as marketers eagerly search for new ways to increase engagement and loyalty. The key to a sustainable and profitable direct mail program is commitment to a regular testing schedule and adopting a results mindset. Even if your current direct mail campaigns are successful, remember the words of David Ogilvy: continue to refine the elements that make your campaigns great and always watch for opportunities to improve.