The SMART measures

In 1966, Rocchio gave a derivation of two overall indices of merit based on recall and precision. They were proposed for the evaluation of retrieval systems which ranked documents, and were designed to be independent of cut-off.

The first of these indices is normalised recall. It roughly measures the effectiveness of the ranking in relation to the best possible and worst possible ranking. The situation is illustrated in Figure 7.9 for 25 documents where we plot on the y-axis and the ranks on the x-axis.

Normalised recall (Rnorm) is the area between the actual case and the worst as a proportion of the area between the best and the worst. If n is the number of relevant documents, and ri the rank at which the ith document is retrieved, then the area between the best and actual case can be shown to be (after a bit of algebra):

(see Salton[23], page 285).

A convenient explicit form of normalised recall is:

where N is the number of documents in the system and N - n the area between the best and the worst case (to see this substitute ri = N - i + 1 in the formula for Ab - Aa). The form ensures that Rnorm lies between 0 (for the worstcase) and 1 (for the best case).