MEASURING THE EFFECTIVENESS OF ASSIGNING CODES: RECALL AND PRECISION

Recall and precision are two measurements that are useful for determining the appropriateness of a set of assigned codes or keywords. The case study on coding news stories, for instance, assigns many codes to news stories. Recall and precision can be used to evaluate these assignments.

Recall answers the question: How many of the correct codes did MBR assign to the story? It is the ratio of codes assigned by MBR that are correct (as verified by editors) to the total number of correct codes on the story. If MBR assigns all available codes to every story, then recall is 100 percent because the correct codes all get assigned, along with many other irrelevant codes. If MBR assigns no codes to any story, then recall is 0 percent.

Precision answers the question: How many of the codes assigned by MBR were correct? It is the percentage of correct codes assigned by MBR to the total number of codes assigned by MBR. Precision is 100 percent when MBR assigns only correct codes to a story. It is close to 0 percent when MBR assigns all codes to every story.

Neither recall nor precision individually give the full story of how good the classification is. Ideally, we want 100 percent recall and 100 percent precision. Often, it is possible to trade off one against the other. For instance, using more neighbors increases recall, but decreases precision. Or, raising the threshold increases precision but decreases recall. Table 8.5 gives some insight into these measurements for a few specific cases.

Table 8.5 Examples of Recall and Precision

The original codes assigned to the stories by individual editors had a recall of 83 percent and a precision of 88 percent with respect to the validated set of correct codes. For MBR, the recall was 80 percent and the precision 72 percent. However, Table 8.6 shows the average across all categories. MBR did significantly better in some of the categories.

MEASURING THE EFFECTIVENESS OF ASSIGNING CODES: RECALL AND PRECISION (continued)

Table 8.6 Recall and Precision Measurements by Code Category

The variation in the results by category suggests that the original stories used for the training set may not have been coded consistently. The results from MBR can only be as good as the examples chosen for the training set. Even so, MBR performed as well as all but the most experienced editors.

Building a Distance Function One Field at a Time

It is easy to understand distance as a geometric concept, but how can distance be defined for records consisting of many different fields of different types? The answer is, one field at a time. Consider some sample records such as those shown in Table 8.7.

Figure 8.6 illustrates a scatter graph in three dimensions. The records are a bit complicated, with two numeric fields and one categorical. This example shows how to define field distance functions for each field, then combine them into a single record distance function that gives a distance between two records.

Table 8.7 Five Customers in a Marketing Database

$120,000 $100,000 $80,000 $60,000 $40,000 $20,000 $0

Female Male

25 30 35 40 45 50 55 60 Age

Figure 8.6 This scatter plot shows the five records from Table 8.7 in three dimensions-age, salary, and gender-and suggests that standard distance is a good metric for nearest neighbors.

The four most common distance functions for numeric fields are:

Absolute value of the difference: A-B

Square of the difference: (A-B)2

Normalized absolute value: A-B/(maximum difference)

Absolute value of difference of standardized values: (A - mean)/(stan-dard deviation) - (B - mean)/(standard deviation) which is equivalent to (A - B)/(standard deviation)

The advantage of the normalized absolute value is that it is always between 0 and 1. Since ages are much smaller than the salaries in this example, the normalized absolute value is a good choice for both of them-so neither field will dominate the record distance function (difference of standardized values is also a good choice). For the ages, the distance matrix looks like Table 8.8.

Table 8.8 Distance Matrix Based on Ages of Customers