Concepts
Similar pages
| Similarity |
Page |
Snapshot |
| 174 |
one unit of precision for an increase of one unit of recall,but will not sacrifice another unit of precision for a further unit increase in recall,i
...R 1,P 1 >R,P but R 1,P >R 2,P 1 We conclude that the interval between R 1 and R exceeds the interval between P and P 1 whereas the interval between R 1 and R 2 is smaller
...Finally,we incorporate into our measurement procedure the fact that users may attach different relative importance to precision and recall
...Definition 6
...Can we find a function satisfying all these conditions?If so,can we also interpret it in an intuitively simple way?The answer to both these questions is yes
...The scale functions are therefore,[[Phi]]1 P [[alpha]]1 P,and [[Phi]]2 R 1 [[alpha]]1 R
...We now have the effectiveness measure
... |
| 10 |
In the past there has been much debate about the validity of evaluations based on relevance judgments provided by erring human beings
...Effectiveness and efficiency Much of the research and development in information retrieval is aimed at improving the effectiveness and efficiency of retrieval
... |
| 154 |
Composite measures Dissatisfaction in the past with methods of measuring effectiveness by a pair of numbers e
...S P R This is simply related to a measure suggested by Borko BK P R 1 More complicated ones are Vickery s measure V can be shown to be a special case of a general measure which will be derived below
...Some single number measures have derivations which can be justified in a rational manner
...The Swets model As early as 1963 Swets [12]showed that the suggested modifications were in fact simply related to an alternative measure already suggested by Swets
...It is interesting that although the Swets model is theoreticallyattractive |
| 146 |
There has been much debate in the past as to whether precision and recall are in fact the appropriate quantities to use as measures of effectiveness
...1 the most commonly used pair;2 fairly well understood quantities
...The final question How to evaluate?has a large technical answer
...Before proceeding to the technical details relating to the measurement of effectiveness it is as well to examine more closely the concept of relevance which underlies it
...Relevance Relevance is a subjective notion
... |
| 188 |
In basing a theory of evaluation on the theory of measurement,is it possible to devise a measure of effectiveness not starting with precision and recall but simply with the set of relevant documents and the set of retrieved documents?If so,can we generalise such a measure to take account of degree of relevance?An alternative derivation of an E type measure could be done in terms of recall and fallout
...Up to now the measurement of effectiveness has proved fairly intractable to statistical analysis
...I think the Robertson model described in Chapter 7 goes some way to being considered as a reasonable statistical model
...There may be laws of retrieval such as the well known trade off between precision and recall that are worth establishing either empirically or by theoretical argument
...6
...There is a need for more intensive research into the problems of what to use to represent the content of documents in a computer
...Information retrieval systems,both operational and experimental,have been keyword based
...The major reason for this rather simple minded approach to document retrieval is a very good one
... |
| 168 |
Foundation Problems of measurement have arisen in physics,psychology,and more recently,the social sciences
...The problems of measurement in information retrieval differ from those encountered in the physical sciences in one important aspect
...The next three sections are substantially the same as those appearing in my paper:Foundations of evaluation,Journal of Documentation,30,365 373 1974
...no reason why we cannot postulate a particular ordering,or,to put it more mildly,why we can not show that a certain model for the measurement of effectiveness has acceptable properties
...1 all properties ascribed are consistent;2 they bring out into the open all the assumptions made in measuring effectiveness;3 each property has an acceptable interpretation;4 the model leads to a plausible measure of effectiveness
...It is as well to point out here that it does not lead to a uniquemeasure,but it does show that certain classes of measures can beregarded as being equivalent
... |
| 176 |
conjoint structure
...The analysis is not limited to the two factors precision and recall,it could equally well be carried out for say the pair fallout and recall
...Presentation of experimental results In my discussion of micro,macro evaluation,and expected search length,various ways of averaging the effectiveness measure of the set of queries arose in a natural way
...In this section the discussion will be restricted to single number measures such as a normalised symmetric difference,normalised recall,etc
...The measurements we have therefore are Za Q 1,Za Q 2,... |
| 169 |
The model We start by examining the structure which it is reasonable to assume for the measurement of effectiveness
...If R is the set of possible recall values and P is the set of possible precision values then we are interested in the set R x P with a relation on it
...Definition 1
...1 Connectedness:either e 1 >e 2 or e 2 >e 1 2 Transitivity:if e 1 >e 2 and e 2 >e 3 then e 1 >e 3 We insist that if two pairs can be ordered both ways then R 1,P 1 R 2,P 2,i
...We now turn to a second condition which is commonly called independence
...Definition 2
...All we are saying here is,given that at a constant recall precision we find a difference in effectiveness for two values of precision recall then this difference cannot be removed or reversed by changing the constant value
...We now come to a condition which is not quite as obvious as the preceding ones
... |
| 175 |
To facilitate interpretation of the function,we transform according to [[alpha]]1 ß2 1,and find that [[partialdiff]]E [[partialdiff]]R [[partialdiff]]E [[partialdiff]]P when P R ß...E now gives rise to the following special cases:1 When [[alpha]]1 2 ß1 E A [[Delta]]B A B,a normalised symmetric difference between sets A and B A [[Delta]]B A [[union]]B A [[intersection]]B
...2 E >1 R when [[alpha]]>0 ß>,which corresponds to a user who attaches no important to precision
...3 E >1 P when [[alpha]]>1 ß>0,which corresponds to a user who attaches no importance to recall
...It is now a simple matter to show that certain other measures given in the literature are special cases of the general form E
...which is the measure recommended by Heine [3]...One final example is the measure suggested by Vickery in 1965 which was documented by Cleverdon et al
...which is Vickery s measure apart from a scale factor of 100
...To summarise,we have shown that it is reasonable to assume thateffectiveness in terms of precision and recall determines an additive |
|
|
