Since becoming a DM consultant last year, following a ten-year career at The Reader’s Digest Association, I was surprised how many companies lacked rigorous discipline for test planning and analysis. The most common reason for not employing statistically sound testing and analysis methods? Unfamiliarity of appropriate methods and the ramifications of not employing those methods!

Most marketers acknowledge the importance of adequate test panel sizes, however, they lack the understanding of the parameters used to calculate the sample size and analyze the results. To make the planning and analysis process as easy as possible, I developed a software tool called The Plan-alyzer(c) , and it’s available free to industry professionals. Safely download it free from the Drake Direct website at www.drakedirect.com. No strings attached. It’s compatible with Window 95 and Windows 98.

The steps for proper test planning and analysis and how The Plan-alyzer(c) can help you perform these steps are:

1) Determining Test Panel Sample Sizes: Without adequate sample sizes, the error variance associated with tests will be so great, results will be unreliable. They will not resemble what you can expect when you roll-out a new test format to a larger audience. If your testing budget is not large enough to test more names per panel, reduce the number of test panels, not the number of names tested. The Plan-alyzer(c) helps you determine: o the appropriate sample size for a single test to ensure its error variance does not exceed a certain threshold and o the sample sizes required for two test panels to ensure the difference is reliable and not due to chance.

2) Gauging Test Panel Error Rates: No matter how large your test sample, you will never receive the exact results in roll-out. Why? Every test response rate has a certain level of error variance. Placing bounds around your sample estimates will allow you to be confident the true response rate will fall within those bounds. The Plan-alyzer(c) makes it easy to place confidence bounds around a sample estimate. Then, you can run various profit calculations using the upper and lower bounds of the interval, thereby determining a “best and worse case” profit scenario prior to roll-out.

3) Assessing the Strength of One Test Panel against Another: Too often, marketers only visually examine the difference between tests, not taking into account the error variance associated with the test response rates. Depending upon the error variance in your test results, you could be making incorrect, costly marketing decisions. Performing a hypothesis test on your marketing test panels takes the error variance into account during assessment. With The Plan-alyzer(c) you can easily gauge the significance of the difference between two test panels. In fact, The Plan-alyzer(c) will not only tell you whether the new test format has beaten the control, but with how much confidence.

Proper test planning and analysis is the single most important analytical topic you can understand in today’s competitive market. Don’t risk overlooking a “test winner” or being mislead by a “false test winner.” Instead, let The Plan-alyzer(c) help you maximize the power of your marketing tests.