| Similarity |
Page |
Snapshot |
| 10 |
In the past there has been much debate about the validity of evaluations based on relevance judgments provided by erring human beings
...Effectiveness and efficiency Much of the research and development in information retrieval is aimed at improving the effectiveness and efficiency of retrieval
... |
| 146 |
There has been much debate in the past as to whether precision and recall are in fact the appropriate quantities to use as measures of effectiveness
...1 the most commonly used pair;2 fairly well understood quantities
...The final question How to evaluate?has a large technical answer
...Before proceeding to the technical details relating to the measurement of effectiveness it is as well to examine more closely the concept of relevance which underlies it
...Relevance Relevance is a subjective notion
... |
| 145 |
automatic and interactive retrieval system?Studies to gauge this are going on but results are hard to interpret
...It should be apparent now that in evaluating an information retrieval system we are mainly concerned with providing data so that users can make a decision as to 1 whether they want such a system social question and 2 whether it will be worth it
...The second question what to evaluate?boils down to what can we measure that will reflect the ability of the system to satisfy the user
...1 The coverage of the collection,that is,the extent to which the system includes relevant matter;2 the time lag,that is,the average interval between the time the search request is made and the time an answer is given;3 the form of presentation of the output;4 the effort involved on the part of the user in obtaining answers to his search requests;5 the recall of the system,that is,the proportion of relevant material actually retrieved in answer to a search request;6 the precision of the system,that is,the proportion of retrieved material that is actually relevant
...It is claimed that 1 4 are readily assessed
... |
| 155 |
and links IR measurements to a ready made and well developed statistical theory,it has not found general acceptance amongst workers in the field
...Before proceeding to an explanation of the Swets model,it is as well to quote in full the conditions that the desired measure of effectiveness is designed to meet
...A desirable measure of retrieval performance would have the following properties:First,it would express solely the ability of a retrieval system to distinguish between wanted and unwanted items that is,it would be a measure of effectiveness only,leaving for separate consideration factors related to cost or efficiency
...He then goes on to claim that The measure I proposed [in 1963],one drawn from statistical decision theory,has the potential [my italics]to satisfy all four desiderata
...To arrive at the measure,we must first discuss the underlying model
...Recall an estimate of the conditional probability that an item will be retrieved given that it is relevant [we denote this P B A]...Precision an estimate of the conditional probability that an item will be relevant given that it is retrieved [i
...Fallout an estimate of the conditional probability that an item will be retrieved given that it is non relevant [i
... |
| 148 |
relevant to an information need if and only if it contains at least one sentence which is relevant to that need
...Earlier on I stated that this notion of relevance was only of limited use at the moment
...Saracevic [8]has summarised some of the more recent work on probabilistic interpretations of relevance
...Precision and recall,and others We now leave the speculations about relevance and return to the promised detailed discussion of the measurement of effectiveness
...It is helpful at this point to introduce the famous contingency table which is not really a contingency table at all
... |
| 24 |
is no doubt that stems rather than ordinary word forms are more effective Carroll and Debruyn [19]...In the next sections I shall give a simple discussion of the kind of frequency information that may be used to weight document descriptors and explain the use of automatically constructed term classes to aid retrieval
...Index term weighting Traditionally the two most important factors governing the effectiveness of an index language have been thought to be the exhaustivity of indexing and the specificity of the index language
...For any document,indexing exhaustivity is defined as the number of different topics indexed,and the index language specificity is the ability of the index language to describe topics precisely
...It is of some importance to be able to quantify the notions of indexing exhaustivity and specificity because of the predictable effect they have on retrieval effectiveness
...Quite a few people Sparck Jones [22,23],have attempted to relate these two factors to document collection statistics
... |
