Finding Your Confidence Interval

Finding Your Confidence Interval

By Perry D. Drake

The following article, written by Perry D. Drake, appeared in Inside Direct Mail, September 1999. It provides direct marketers with a means of better interpreting their testing results.

When conducting marketing tests using samples, measures such as the percent responding or the payment rate are the “measures of concern.”

Depending on the results of the sample estimates, in conjunction with the marketing cost and revenue figures, you will make a business decision of whether or not to roll-out with a particular list of names, product offering or promotion.

Confidence in your sample estimates is critical to making correct marketing decisions. Unfortunately, regardless of how good your sample is or how little things change from test to roll-out, with almost 100 percent certainty, you will never receive the exact same result in roll-out that you did in test.

The problem is an error associated with sample estimates.

If, for example, the test reveals a four-percent response rate to a particular offer, you will not receive that response rate in roll-out regardless of having a large, and properly drawn sample. The reason: A sample is just that–a sample. Sample estimates have a certain level of associated error variance. The higher the error variance associated with your sample estimates, the more likely your test results will be “off” from what you can expect in roll-out.

But, this will not prevent you from making accurate marketing decisions based on these sample estimates.

In this article, you will learn how to place bounds around a test response rate, allowing you to determine a range in which the true response rate can lie in a roll-out situation. Once calculated, you can run various profit calculations using the upper and lower bounds of the interval to determine a best and worse case profit scenario for the roll-out.

Constructing the Confidence Interval
A confidence interval constructed around a single response rate is primarily used to assess a new list test in terms of its potential as a “new name” generator or to assess a new product in terms of its potential for a large scale roll-out.

To calculate a confidence interval around a test proportion, the following information is required:
  • The test response rate, which we’ll label p. This is response rate obtained for the test (i.e. percent responders, payment rate, percent telemarketing hits.).

  • The sample size of the test, which we’ll label n. This is the number of names tested. NOTE: In order to use this formula, the sample size when multiplied by the sample response rate and when multiplied by one minus the sample response rate, must be greater than or equal to five.

  • The desired confidence level. A confidence interval constructed around the test response rate “p” will guarantee, with your specified level of confidence, that the true population proportion obtained in roll-out will fall within those bounds. For example, constructing an interval around your test response rate with a 90-percent confidence level will guarantee that the true response rate you can expect in roll-out will fall within the bounds of the confidence interval with a 90-percent probability.
NOTE: It is strongly advised that you never construct a confidence interval with less than a 90-percent confidence level. Keep in mind, that with a 90-percent confidence level there is a 10-percent chance (100 percent minus 90 percent) that the true response rate you can expect in the roll-out will not fall within the bounds of the constructed interval. Do you really want to be subjected to more than a 10-percent risk of this happening? Dropping the confidence level below 90-percent will cause you to assume more risk than you should be willing to accept in being misled by the resulting bounds of your confidence interval. As a result, you may make an incorrect marketing decision.

The formula used to calculate the lower and upper bounds of a confidence interval is:
Lower bound = “p” minus “z” times the standard deviation of the test response
Upper bound = “p” plus “z” times the standard deviation of the test response
Where the standard deviation of the test response rate is equal to the square root of the following calculation: the test response rate times one minus the test response rate divided by the sample size

And, where z equals 1.645, 1.96 or 2.575 for a 90-, 95- or 99-percent confidence level, respectively.

To illustrate, assume the marketing director at ACME Direct has conducted a new product test to 10,000 names from the primary customer segment of the ACME Direct database. The test results for this new product test yielded an order rate of 3.42 percent.

To assess the potential for this new product in a roll-out, the marketing director decides to construct a confidence interval that is guaranteed to contain the true response rate he can expect in roll-out with 95-percent confidence.

Using the previously mentioned formula we first calculate the standard deviation of the test response rate as follows:
 Standard deviation of the test response rate= Square root of ((.0342 x (1 – .0342))/10,000)
= Square root of (.0000033)
= .0018165

Next, we calculate the lower and upper bounds of the 95-percent confidence interval using z = 1.96:

 Lower bound

= .0342 – (1.96) x (.0018165)
= .0306

 Upper bound

= .0342 + (1.96) x (.0018165)
= .0378

The resulting 95-percent confidence interval is 3.06 percent to 3.78 percent. The marketing director can be 95-percent confident, should he decide to roll-out with the new product, the response rate will be no less than 3.06 percent, and no higher than 3.78 percent. He can use the lower bound to determine the worse case scenario in terms of profitability. Upon examination of this profit scenario, the marketing director will make his decision.

The confidence level selected depends on the risk you are willing to take in the resulting confidence interval not containing the response rate to expect in a roll-out. If the risk is quite high, set the confidence level at 99 percent. If the risk is minimal to none, use industry standard levels of 90 percent to 95 percent. Base the “risk” on the cost figures associated with marketing the new product or offer. When in doubt, choose a 95 percent confidence level.

Interpreting the Confidence Interval Assume the marketing director had previously determined that he needed at least a 3.25-percent response rate to generate 10-percent profit after overhead for this new product–his objective. The lower bound of the 90-percent confidence interval was 3.06 percent. Therefore, if the marketing director rolls out with this new product offering, there is a chance that the response rate achieved will be lower than the minimum he requires. What should he do? He will:
  • Calculate the profit assuming he receives only a 3.06-percent response rate (the lower bound of the confidence interval) in roll-out.

  • Take into consideration the fact that the upside potential (3.78 percent) far outweighs the downside potential (3.06 percent) given he needs only a 3.25-percent response rate to meet his goal of 10-percent profit after overhead.

  • Calculate a more conservative estimate of the worse case scenario, in terms of the response rate to expect in roll-out, by constructing a 99-percent confidence interval. Then assess the lower bound of this interval in terms of profitability.

Using this information, the marketing director can make a wise marketing decision. Just remember, confidence intervals will not give you a definitive answer to your question but will help you in assessing the test results by providing you with various scenarios based on the amount of error associated with your sample test results.

Another option available to direct marketers in making a more informed decision is testing more names. Testing more names minimizes the error associated with the testing results. While most list and format test quantities begin at 5,000 names, testing a larger sample can often give you a better picture of the responsiveness of the list or format. Testing more names minimizes the error associated with the testing results.

If your testing budget won’t allow you to test more names, cut back on the number of test panels, not on the number of names tested per panel. Five test panels, for example each with an appropriate test panel size, will benefit your company far greater that ten test panels with too few names promoted in each.