Figure 3.2 shows another example of confusion caused by aggregation. Sales appear to be down in October compared to August and September. The picture comes from a business that has sales activity only on days when the financial markets are open. Because of the way that weekends and holidays fell in 2003, October had fewer trading days than August and September. That fact alone accounts for the entire drop-off in sales.

In the previous examples, aggregation led to confusion. Failure to aggregate to the appropriate level can also lead to confusion. In one case, data provided by a charitable organization showed an inverse correlation between donors likelihood to respond to solicitations and the size of their donations. Those more likely to respond sent smaller checks. This counterintuitive finding is a result of the large number of solicitations the charity sent out to its supporters each year. Imagine two donors, each of whom plans to give $500 to the charity. One responds to an offer in January by sending in the full $500 contribution and tosses the rest of the solicitation letters in the trash. The other sends a $100 check in response to each of five solicitations. On their annual income tax returns, both donors report having given $500, but when seen at the individual campaign level, the second donor seems much more responsive. When aggregated to the yearly level, the effect disappears.

Learning Things That Are True, but Not Useful

Although not as dangerous as learning things that arent true, learning things that arent useful is more common.

Sales by Month (2003)

43500

43000

42500

42000

41500

41000

40500

40000

August September

Figure 3.2 Did sales drop off in October?

October

Learning Things That Are Already Known

Data mining should provide new information. Many of the strongest patterns in data represent things that are already known. People over retirement age tend not to respond to offers for retirement savings plans. People who live where there is no home delivery do not become newspaper subscribers. Even though they may respond to subscription offers, service never starts. For the same reason, people who live where there are no cell towers tend not to purchase cell phones.

Often, the strongest patterns reflect business rules. If data mining discovers that people who have anonymous call blocking also have caller ID, it is perhaps because anonymous call blocking is only sold as part of a bundle of services that also includes caller ID. If there are no sales of certain products in a particular location, it is possible that they are not offered there. We have seen many such discoveries. Not only are these patterns uninteresting, their strength may obscure less obvious patterns.

Learning things that are already known does serve one useful purpose. It demonstrates that, on a technical level, the data mining effort is working and the data is reasonably accurate. This can be quite comforting. If the data and the data mining techniques applied to it are powerful enough to discover things that are known to be true, it provides confidence that other discoveries are also likely to be true. It is also true that data mining often uncovers things that ought to have been known, but were not; that retired people do not respond well to solicitations for retirement savings accounts, for instance.

Learning Things That Cant Be Used

It can also happen that data mining uncovers relationships that are both true and previously unknown, but still hard to make use of. Sometimes the problem is regulatory. A customers wireless calling patterns may suggest an affinity for certain land-line long-distance packages, but a company that provides both services may not be allowed to take advantage of the fact. Similarly, a customers credit history may be predictive of future insurance claims, but regulators may prohibit making underwriting decisions based on it.

Other times, data mining reveals that important outcomes are outside the companys control. A product may be more appropriate for some climates than others, but it is hard to change the weather. Service may be worse in some regions for reasons of topography, but that is also hard to change.

Sometimes it is only a failure of imagination that makes new information appear useless. A study of customer attrition is likely to show that the strongest predictors of customers leaving is the way they were acquired. It is too late to go back and change that for existing customers, but that does not make the information useless. Future attrition can be reduced by changing the mix of acquisition channels to favor those that bring in longer-lasting customers.

The data mining methodology is designed to steer clear of the Scylla of learning things that arent true and the Charybdis of not learning anything useful. In a more positive light, the methodology is designed to ensure that the data mining effort leads to a stable model that successfully addresses the business problem it is designed to solve.

Hypothesis Testing

Hypothesis testing is the simplest approach to integrating data into a companys decision-making processes. The purpose of hypothesis testing is to substantiate or disprove preconceived ideas, and it is a part of almost all data mining endeavors. Data miners often bounce back and forth between approaches, first thinking up possible explanations for observed behavior (often with the help of business experts) and letting those hypotheses dictate the data to be analyzed. Then, letting the data suggest new hypotheses to test.

Hypothesis testing is what scientists and statisticians traditionally spend their lives doing. A hypothesis is a proposed explanation whose validity can be tested by analyzing data. Such data may simply be collected by observation or generated through an experiment, such as a test mailing. Hypothesis testing is at its most valuable when it reveals that the assumptions that have been guiding a companys actions in the marketplace are incorrect. For example, suppose that a companys advertising is based on a number of hypotheses about the target market for a product or service and the nature of the responses. It is worth testing whether these hypotheses are borne out by actual responses. One approach is to use different call-in numbers in different ads and record the number that each responder dials. Information collected during the call can then be compared with the profile of the population the advertisement was designed to reach.

Each time a company solicits a response from its customers, whether through advertising or a more direct form of communication, it has an opportunity to gather information. Slight changes in the design of the communication, such as including a way to identify the channel when a prospect responds, can greatly increase the value of the data collected.

By its nature, hypothesis testing is ad hoc, so the term methodology might be a bit strong. However, there are some identifiable steps to the process, the first and most important of which is generating good ideas to test.