Finding Your Confidence IntervalBy Perry D. DrakeThe following article, written by Perry D. Drake, appeared in Inside Direct Mail, September 1999. It provides direct marketers with a means of better interpreting their testing results.When conducting marketing tests using samples, measures such as the percent responding or the payment rate are the “measures of concern.” Depending on the results of the sample estimates, in conjunction with the marketing cost and revenue figures, you will make a business decision of whether or not to roll-out with a particular list of names, product offering or promotion. Confidence in your sample estimates is critical to making correct marketing decisions. Unfortunately, regardless of how good your sample is or how little things change from test to roll-out, with almost 100 percent certainty, you will never receive the exact same result in roll-out that you did in test. The problem is an error associated with sample estimates. If, for example, the test reveals a four-percent response rate to a particular offer, you will not receive that response rate in roll-out regardless of having a large, and properly drawn sample. The reason: A sample is just that–a sample. Sample estimates have a certain level of associated error variance. The higher the error variance associated with your sample estimates, the more likely your test results will be “off” from what you can expect in roll-out. But, this will not prevent you from making accurate marketing decisions based on these sample estimates. In this article, you will learn how to place bounds around a test response rate, allowing you to determine a range in which the true response rate can lie in a roll-out situation. Once calculated, you can run various profit calculations using the upper and lower bounds of the interval to determine a best and worse case profit scenario for the roll-out. Constructing the Confidence Interval A confidence interval constructed around a single response rate is primarily used to assess a new list test in terms of its potential as a “new name” generator or to assess a new product in terms of its potential for a large scale roll-out. To calculate a confidence interval around a test proportion, the following information is required:
The formula used to calculate the lower and upper bounds of a confidence interval is: Lower bound = “p” minus “z” times the standard deviation of the test responseWhere the standard deviation of the test response rate is equal to the square root of the following calculation: the test response rate times one minus the test response rate divided by the sample size And, where z equals 1.645, 1.96 or 2.575 for a 90-, 95- or 99-percent confidence level, respectively. To illustrate, assume the marketing director at ACME Direct has conducted a new product test to 10,000 names from the primary customer segment of the ACME Direct database. The test results for this new product test yielded an order rate of 3.42 percent. To assess the potential for this new product in a roll-out, the marketing director decides to construct a confidence interval that is guaranteed to contain the true response rate he can expect in roll-out with 95-percent confidence. Using the previously mentioned formula we first calculate the standard deviation of the test response rate as follows: | ||||||||||||||||
| ||||||||||||||||
| The resulting 95-percent confidence interval is 3.06 percent to 3.78 percent. The marketing director can be 95-percent confident, should he decide to roll-out with the new product, the response rate will be no less than 3.06 percent, and no higher than 3.78 percent. He can use the lower bound to determine the worse case scenario in terms of profitability. Upon examination of this profit scenario, the marketing director will make his decision. The confidence level selected depends on the risk you are willing to take in the resulting confidence interval not containing the response rate to expect in a roll-out. If the risk is quite high, set the confidence level at 99 percent. If the risk is minimal to none, use industry standard levels of 90 percent to 95 percent. Base the “risk” on the cost figures associated with marketing the new product or offer. When in doubt, choose a 95 percent confidence level. Interpreting the Confidence Interval Assume the marketing director had previously determined that he needed at least a 3.25-percent response rate to generate 10-percent profit after overhead for this new product–his objective. The lower bound of the 90-percent confidence interval was 3.06 percent. Therefore, if the marketing director rolls out with this new product offering, there is a chance that the response rate achieved will be lower than the minimum he requires. What should he do? He will:
Using this information, the marketing director can make a wise marketing decision. Just remember, confidence intervals will not give you a definitive answer to your question but will help you in assessing the test results by providing you with various scenarios based on the amount of error associated with your sample test results. Another option available to direct marketers in making a more informed decision is testing more names. Testing more names minimizes the error associated with the testing results. While most list and format test quantities begin at 5,000 names, testing a larger sample can often give you a better picture of the responsiveness of the list or format. Testing more names minimizes the error associated with the testing results. If your testing budget won’t allow you to test more names, cut back on the number of test panels, not on the number of names tested per panel. Five test panels, for example each with an appropriate test panel size, will benefit your company far greater that ten test panels with too few names promoted in each. | ||||||||||||||||