| Similarity |
Page |
Snapshot |
| 164 |
The SMART measures In 1966,Rocchio gave a derivation of two overall indices of merit based on recall and precision
...The first of these indices is normalised recall
...Normalised recall Rnorm is the area between the actual case and the worst as a proportion of the area between the best and the worst
...see Salton [23],page 285
...A convenient explicit form of normalised recall is:where N is the number of documents in the system and N n the area between the best and the worst case to see this substitute ri N i 1 in the formula for Ab Aa
... |
| 176 |
conjoint structure
...The analysis is not limited to the two factors precision and recall,it could equally well be carried out for say the pair fallout and recall
...Presentation of experimental results In my discussion of micro,macro evaluation,and expected search length,various ways of averaging the effectiveness measure of the set of queries arose in a natural way
...In this section the discussion will be restricted to single number measures such as a normalised symmetric difference,normalised recall,etc
...The measurements we have therefore are Za Q 1,Za Q 2,... |
| 166 |
differ considerably from those which the user feels are pertinent Senko [21]...Fourthly,whereas Cooper has gone to some trouble to take account of the random element introduced by ties in the matching function,it is largely ignored in the derivation of Pnorm and Rnorm
...One further comment of interest is that Robertson 15 has shown that normalised recall has an interpretation as the area under the Recall Fallout curve used by Swets
...Finally mention should be made of two similar but simpler measures used by the SMART system
...and do not take into account the collection size N,n is here the number of relevant documents for the particular test query
...A normalised symmetric difference Let us now return to basics and consider how it is that users could simply measure retrieval effectiveness
... |
| 10 |
In the past there has been much debate about the validity of evaluations based on relevance judgments provided by erring human beings
...Effectiveness and efficiency Much of the research and development in information retrieval is aimed at improving the effectiveness and efficiency of retrieval
... |
| 163 |
normalising the ESL by a factor proportional to the expected number of non relevant documents collected for each relevant one
...which has been called the expected search length reduction factor by Cooper
...where 1 R is the total number of documents in the collection relevant to q;2 I is the total number of documents in the collection non relevant to q;3 S is the total desired number of documents relevant to q
...The explicit form for ESL was given before
...which is known as the mean expected search length reduction factor
...Within the framework as stated at the head of this section this final measure meets the bill admirably
...For a further defence of its subjective nature see Cooper [1]... |
| 165 |
In an analogous manner normalised precision is worked out
...The calculation of the areas is a bit more messy but simple to do see Salton [23],page 298
...The log function appears as a result of approximating [[Sigma]]1 r by its continuous analogue [[integral]]1 r dr,which is log r constant
...The area between the worst and best case is obtained in the same way as before using the same substitution,and is:The explicit form,with appropriate normalisation,for normalised precision is therefore:Once again it varies between 0 worst and 1 best
...A few comments about these measures are now in order
... |
| 146 |
There has been much debate in the past as to whether precision and recall are in fact the appropriate quantities to use as measures of effectiveness
...1 the most commonly used pair;2 fairly well understood quantities
...The final question How to evaluate?has a large technical answer
...Before proceeding to the technical details relating to the measurement of effectiveness it is as well to examine more closely the concept of relevance which underlies it
...Relevance Relevance is a subjective notion
... |
| 180 |
effectiveness can be calculated to infinite precision we may be insisting on a difference when in fact it only occurs in the tenth decimal place
...Finally,although I have just explained the use of the sign test in terms of single number measures,it is also used to detect a significant difference between precision recall graphs
...Bibliographic remarks Quite a number of references to the work on evaluation have already been given in the main body of the chapter
...Buried in the report by Keen Digger [32]Chapter 16 is an excellent discussion of the desirable properties of any measure of effectiveness
...A parameter which I have mentioned in passing but which deserves closer study in generality
...The trade off between precision and recall has for a long time been the subject of debate
...Guazzo [39]describe an approach to the measurement of retrieval effectiveness based on information theory
...The notion of relevance has at all times attracted much discussion
... |
| 156 |
He accepts the validity of measuring the effectiveness of retrieval by a curve either precision recall or recall fallout generated by the variation of some control variable [[lambda]]e
...In the simplest case we assume that the variable [[lambda]]is distributed normally on the set of relevant and non relevant documents
...The usual set up in IR is now to define a decision rule in terms of [[lambda]],to determine which documents are retrieved the acceptance criterion
... |
