Building the Data Mining

Environment

In the Big Rock Candy Mountains, Theres a land thats fair and bright, Where the handouts grow on bushes And you sleep out every night. Where the boxcars all are empty And the sun shines every day And the birds and the bees And the cigarette trees The lemonade springs Where the bluebird sings In the Big Rock Candy Mountains.

Twentieth century hoboes had a vision of utopia, so why not twenty-first century data miners? For us, the vision is one of a company that puts the customer at the center of its operations and measures its actions by their effect on long-term customer value. In this ideal organization, business decisions are based on reliable information distilled from vast quantities of customer data. Needless to say, data miners-the people with the skills to turn all that data into the information needed to run the company-are held in great esteem.

This chapter starts with a utopian vision of a truly customer-centric organization with the ideal data mining environment to produce the information on which all decisions are based. Having a description of what the ideal data mining environment would look like is helpful for establishing more realistic near term goals. The chapter then goes on to look at the various components of the data mining environment-the staff, the data mining infrastructure, and the data mining software itself. Although we may not be able to achieve all elements of the utopian vision, we can use the vision to help create an environment suitable for successful data mining work.

A Customer-Centric Organization

Despite the familiar cliche that the customer is king, in most companies customers are not treated much like royalty. One reason is that most businesses are not organized around customers; they are organized around products. Supermarkets, for example, have long been able to track the inventory levels of tens of thousands of products in order to keep the shelves well stocked, and they are able to calculate the profit margin on any item. But, until recently, these same stores knew nothing about individual customers-not their names, nor how many trips per month they make, nor what time of day they tend to shop, nor whether they use coupons, nor if they have children, nor what percent of the households shopping is done in this store, nor how close they live-nothing. We dont mean to pick on supermarkets. Banks have been organized around loans; telephone companies have been organized around switches; airlines have been organized around operations. None have known much (or cared much) about customers.

In all of these industries, technology now makes it possible to shift the focus to customers. Such a shift is not easy; in fact, it is nothing short of revolutionary. By combining point-of-sale scanner data with a loyalty card program, a grocery retailer can, with a lot of effort, learn who is buying what and when they buy it, which customers are price-sensitive and which ones like to try new products, which ones like to bake from scratch and which ones prefer prepared meals, and so on. A telephone company can figure out who is making business calls and who is primarily chatting with friends. An online music store can make individualized recommendations of new music.

The harder challenge is being able to make effective use of this new ability to see customers in data. A truly customer-centric organization would be happy to continue offering an unprofitable service if the customers who use the loss-generating service spend more in other areas and therefore increase the profitability of the company as a whole. A customer-centric company does not have to ask the same questions every time a customer calls in. A customer-centric company judges a marketing campaign on the value customers generate over their lifetimes rather than on the initial response rate.

Becoming truly customer-centric means changing the corporate culture and the way everyone from top managers to call-center operators are rewarded. As long as each product line has a manager whose compensation is tied to the amount and margin of product sold, the company will remain focused on products rather than customers. In other words, the company is paying its managers to focus on products, and the managers are doing their jobs. In the ideal customer-centric organization, everyone is rewarded for increasing customer value and understands that this requires learning from each customer

interaction and the ability to use what has been learned to serve customers better. As a result, the company records every interaction with its customers and keeps an extensive historical record of these interactions.

An Ideal Data Mining Environment

The ideal context for data mining is an organization that appreciates the value of information. Bringing together customer data from all of the many places where it is originally collected and putting it into a form suitable for data mining is a difficult and expensive process. It will only happen in an organization that understands how valuable that data is once it can be properly exploited. Information is power. A learning organization values progress and steady improvement; such an organization wants and invests in accurate information. Remember that the producers of information always have real power to determine what data is available and when. They are not passive consumers of a take-it-or-leave-it data warehouse, they have the power to determine what data is available, although collecting such data might mean changing operational procedures.

The Power to Determine What Data Is Available

In the ideal data mining environment, the importance of data analysis is recognized and its results are shared across the organization. Marketing people instinctively regard every campaign as a controlled experiment, even when that means not including some customers in a promising campaign because those customers are part of a control group. Designers of operational systems instinctively keep track of all customer transactions, including nonbillable ones such as customer service inquiries, bank account balance inquiries, or visits to particular sections of the company Web site. Everyone expects that customer interactions from different channels can be identified as involving the same customer, even when some happen at an ATM, some in a bank branch, some over the phone, and some on the Web.

In such an environment, an analyst at a telephone company trying to understand the relationship between quality of wireless telephone service and churn has no trouble getting customer-level data on dropped calls and other failures. The analyst can also readily see a customers purchase history even though some purchases were made in stores, some through the mail-order catalog, and some on the Web. It is similarly easy to determine, for each of a customers calls to customer service, the duration of the call and whether the call was handled by a human representative or stayed in the IVR, and in the latter case, what path was followed through the prompts. Best of all, when the required