BENEFIT (continued)

As used in the medical world, sensitivity is the proportion of true positives among people who get a positive result on a test. In other words, it is the true positives divided by the sum of the true positives and false positives. Sensitivity measures the likelihood that a diagnosis based on the test is correct. Specificity is the proportion of true negatives among people who get a negative result on the test. A good test should be both sensitive and specific. The maximum benefit point is the cutoff that maximizes the average of these two measures. In Chapter 8, these concepts go by the names recall and precision, the terminology used in information retrieval. Recall measures the number of articles on the correct topic returned by a Web search or other text query. Precision measures the percentage of the returned articles that are on the correct topic.

♦ The maximum benefit point corresponds to a decision rule that minimizes the expected loss assuming the misclassification costs are inversely proportional to the prevalence of the target classes.

One way of evaluating classification rules is to assign a cost to each type of misclassification and compare rules based on that cost. Whether they represent responders, defaulters, fraudsters, or people with a particular disease, the rare cases are generally the most interesting so missing one of them is more costly than misclassifying one of the common cases. Under that assumption, the maximum benefit picks a good classification rule.

This table says that if a prospect is contacted and responds, the company makes forty-four dollars. If a prospect is contacted, but fails to respond, the company loses $1. In this simplified example, there is neither cost nor benefit in choosing not to contact a prospect. A more sophisticated analysis might take into account the fact that there is an opportunity cost to not contacting a prospect who would have responded, that even a nonresponder may become a better prospect as a result of the contact through increased brand awareness, and that responders may have a higher lifetime value than indicated by the single purchase. Apart from those complications, this simple profit and loss matrix can be used to translate the response to a campaign into a profit figure. Ignoring campaign overhead fixed costs, if one prospect responds for every 44 who fail to respond, the campaign breaks even. If the response rate is better than that, the campaign is profitable.

WARNINGJIf the cost of a failed contact is set too low, the profit and loss matrix suggests contacting everyone. This may not be a good idea for other reasons. It could lead to prospects being bombarded with innapropriate offers.

Team-Fly®

How the Model Affects Profitability

How does the model whose lift and benefit are characterized by Figure 4.2 affect the profitability of a campaign? The answer depends on the start-up cost for the campaign, the underlying prevalence of responders in the population and on the cutoff penetration of people contacted. Recall that SAC had a budget of $300,000. Assume that the underlying prevalence of responders in the population is 1 percent. The budget is enough to contact 300,000 prospects, or 30 percent of the prospect pool. At a depth of 30 percent, the model provides lift of about 2, so SAC can expect twice as many responders as they would have without the model. In this case, twice as many means 2 percent instead of 1 percent, yielding 6,000 (2% * 300,000) responders each of whom is worth $44 in net revenue. Under these assumptions, SAC grosses $600,000 and nets $264,000 from responders. Meanwhile, 98 percent of prospects or 294,000 do not respond. Each of these costs a dollar, so SAC loses $30,000 on the campaign.

Table 4.4 shows the data used to generate the concentration chart in Figure 4.2. It suggests that the campaign could be made profitable by spending less money to contact fewer prospects while getting a better response rate. Mailing to only 10,000 prospects, or the top 10 percent of the prospect list, achieves a lift of 3. This turns the underlying response rate of 1 percent into a response rate of 3 percent. In this scenario, 3,000 people respond yielding revenue of $132,000. There are now 97,000 people who fail to respond and each of them costs one dollar. The resulting profit is $35,000. Better still, SAC has $200,000 left in the marketing budget to use on another campaign or to improve the offer made in this one, perhaps increasing response still more.

Table 4.4 Lift and Cumulative Gains by Decile

A smaller, better-targeted campaign can be more profitable than a larger and more expensive one. Lift increases as the list gets smaller, so is smaller always better? The answer is no because the absolute revenue decreases as the number of responders decreases. As an extreme example, assume the model can generate lift of 100 by finding a group with 100 percent response rate when the underlying response rate is 1 percent. That sounds fantastic, but if there are only 10 people in the group, they are still only worth $440. Also, a more realistic example would include some up-front fixed costs. Figure 4.3 shows what happens with the assumption that there is a $20,000 fixed cost for the campaign in addition to the cost of $1 per contact, revenue of $44 per response, and an underlying response rate of 1 percent. The campaign is only profitable for a small range of file penetrations around 10 percent.

Using the model to optimize the profitability of a campaign seems more attractive than simply using it to pick whom to include on a mailing or call list of predetermined size, but the approach is not without pitfalls. For one thing, the results are dependent on the campaign cost, the response rate, and the revenue per responder, none of which are known prior to running the campaign. In the example, these were known, but in real life, they can only be estimated. It would only take a small variation in any one of these to turn the campaign in the example above completely unprofitable or to make it profitable over a much larger range of deciles.

($100,000)

($200,000)

($300,000)

($400,000)

($500,000)

($600,000)

Profit by Decile

0% 10% 20% 30%Ч** <40% 50% 60% 70% 80% 90% 100%

$100,000

Figure 4.3 Campaign profitability as a function of penetration.