Scoring is not complete until the scores reside on a customer database somewhere accessible to the software that will be used to select customers for inclusion in marketing campaigns. If Web log or call detail or point-of-sale scanner data needed as a model input resides in flat files on one system, and the customer marketing database resides on another system but the two are accurate as of different dates,this too can be a data processing challenge.

One Example of a Production Data Mining Architecture

Web retailing is an industry that has gone farther than most in routinely incorporating data mining and scoring into the operational environment. Many Web retailers update a customers profile with every transaction and use model scores to determine what to display and what to recommend. The architecture described here is from Blue Martini, a company that supplies software for mining-ready retail Web sites. The example it provides of how data mining can be made an integral part of a companys operations is not restricted to Web retailing. Many companies could benefit from a similar architecture.

Architectural Overview

The Blue Martini architecture is designed to support the differing needs of marketers, merchandisers, and, not least, data miners. As shown in Figure 16.2, it has three modules for three different types of users. For merchandisers, this architecture supports multiple product hierarchies and tools for controlling collections and promotions. For marketers there are tools for making controlled experiments to track the effectiveness of various messages and marketing rules. For data miners, there is integrated modeling software and relief from having to create customer signatures by hand from dozens of different Web server and application logs. The architecture is what Ralph Kimball and Richard Merz would call a data Webhouse, made up of several special-purpose data marts with different schemas, all using common field definitions and shared metadata.

Customers at a Web store interact with pages generated as needed from a database that includes product information and the page templates. The contents of the page are driven by rules. Some of these rules are business rules entered by managers. Others are generated automatically and then edited by professional merchandisers.

Product Hierarchies [-

Business Data Definition Module

I Web Server with logs I

Customer Interaction

Module

Application Server with logs

Figure 16.2 Blue Martini provides a good example of an IT architecture for data mining-driven Web retailing.

Generating pages from a database has many advantages. First it makes it possible to enforce a consistent look and feel across the Web site. Such standard interfaces help customers navigate through the site. Using a database also makes it possible to make global changes quickly, such as updating prices for a sale. Another feature is the ability to store templates in different languages and currencies, so the site can be customized for users in different counties. From the data mining perspective, a major advantage is that all customer interactions are logged in the database.

User interactions are managed through a collection of data marts. Reporting and mining are centered on a customer behavior data mart that includes information derived from the user interaction, product, and business-rule data marts. The complicated extract and transformation logic required to create customer signatures from transaction data is part of the system-a great simplification for anyone who has ever tried massaging Web logs to get information about customers.

Customer Interaction Module

This architecture includes the databases and software needed to support merchandising, customer interaction, reporting, and mining as well as customer-centric marketing in the form of personalization. The Blue Martini system has

three major modules, each with its own data mart. These repositories keep track of the following:

Business rules

Customer and visitor transactions

Customer behavior

The customer behavior data mart, shown in Figure 16.2 as part of the analysis module, is fed by data from the customer interaction module, and it, in turn, supplies rules to both the business data definition module and the customer interaction module.

Merchandising information such as product hierarchies, assortments (families of products that are grouped together for merchandising purposes), and price lists are maintained in the business rules data mart, as is content information such as Web page templates, images, sounds, and video clips. Business rules include personalization rules for greeting named customers, promotion rules, cross-sell rules, and so on. Much of the data mining effort for a retail site goes into generating these rules.

The customer interaction module is the part of the system that touches customers directly by processing all the customer transactions. The customer interaction module is responsible for maintaining users sessions and context. This module implements the actual Web store and collects any data that may be wanted for later analysis. The customer transaction data mart logs business events such as the following:

Customer adds an item to the basket.

Customer initiates check-out process.

Customer completes check-out process.

Cross-sell rule is triggered, and recommendation is made.

Recommended link is followed.

The customer interaction module supports marketing experiments by implementing control groups and keeping track of multiple rules. It has detailed knowledge of the content it serves and can track many things that are not tracked in the Web server logs. The customer interaction module collects data that allows both products and customers to be tracked over time.

Analysis Module

The database that supports the customer interaction module, like most online transaction processing systems, is a relational database designed to support quick transaction processing. Data destined for the analytic module must be extracted and transformed to support the structures suitable for mining and reporting. Data mining requires flat signature tables with one row per customer